Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Closing this for now -- happy to revive it if anyone has other ideas or
wants to run with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Yeah @BryanCutler basically I was just modifying the existing test to
include an artifact with a classifier. If you just add one to the first line in
this test in `SparkSubmitUtilsSuite` I think
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
Oh, I see. I would have expected it to resolve those dependencies too, but
it's hard to tell from the Ivy code what it will do. Was this part of a unit
test you added here, or just some
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Yeah, I think it's a little bigger than that -- the test suggests that it
won't load dependencies of an artifact with a classifier, which is better than
nothing maybe, but still broken enough that
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
>There's a smaller corner case that doesn't quite seem to work, and that is
adding both a classified and unclassified version of the same artifact,
Yeah, I'm not sure why exactly that
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Well it's a little different, now that I've modified the tests to cover the
classifier case. In this case, when the main artifact has a classifier, it
doesn't find the dependency that the test sets
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75936/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75936 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75936/testReport)**
for PR 17416 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Thanks @BryanCutler ! yes that's working better. I can run, for example,
`./bin/spark-shell --packages
edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1` and I can see that it's now
on the
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75936 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75936/testReport)**
for PR 17416 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75914 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75914/testReport)**
for PR 17416 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75914/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75914 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75914/testReport)**
for PR 17416 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
@srowen , I opened https://github.com/srowen/spark/pull/1 in case you
didn't see it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
OK @BryanCutler whatever's easiest for you. Whatever you have working, I
can test and get merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
Sure, I can PR against your branch. I didn't do anything new, just used
the prev version of your `addDependenciesToIvy` method and it seemed to work
for me. I also tried out what would happen
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Thanks @BryanCutler ! so, eh, do you mean you have a working version of
this change? feel free to open it as a PR to master or to this PR or whatever,
whatever's simplest. I'd be pleased to have a
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
@srowen , I finally had some time to look into this and I was able to get
the correct jar on the classpath. The fix was to use the code you had in the
previous commit for
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
@sethah yeah it downloads the right models jar and copies it into the ivy
cache. It just doesn't know that it's what belongs on the classpath. spark.jars
doesn't get the memo and I can see why,
Github user sethah commented on the issue:
https://github.com/apache/spark/pull/17416
@srowen Can you confirm what happens when the jars are not found in your
local m2 cache? Do you still find the `-models` jar in the ivy2 cache?
---
If your project is set up for it, you can reply
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
The essence of the problem is that the `ResolveReport` from `ivy.resolve`
no longer contains any information about the classifier. I can't see why or
what needs to be configured in Ivy to make that
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75389 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75389/testReport)**
for PR 17416 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75389/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75389 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75389/testReport)**
for PR 17416 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Yeah, it clearly downloads the right models .jar file, puts it in the .ivy
cache correctly (I can find it there as expected) but then `spark.jars` isn't
set correctly:
```
...
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
So is the problem that it downloads `stanford-corenlp-3.4.1-models.jar` but
thinks it is `stanford-corenlp-3.4.1.jar`?
It looks like it might be possible to add the classifier to
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Thanks @BryanCutler , here's the current full output after trying to fix
per https://github.com/apache/spark/pull/17416#discussion_r108508229 . Looks
like progress I think but still missed something
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75179/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75179 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75179/testReport)**
for PR 17416 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75179 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75179/testReport)**
for PR 17416 at commit
34 matches
Mail list logo
