Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4780#issuecomment-76507231
So, marking CA as provided but putting it your assembly is contradictory,
but in the end, the right thing is to include CA (and its dependencies) in your
assembly, yes. I was asking whether you mark Spark as provided; it should be.
You're effectively shading CA (and not fastutil); you should also be able
to achieve that through your build rather than bother with source, though, I
don't know how that works in SBT. (`minimizeJar` is a function of Maven's
shading plugin.)
fastutil-in-Spark isn't the issue per se, since indeed Spark doesn't have
it! what it does have is CA.
Your result seems to confirm that the problem is really CA, in the sense
that your app finds Spark's loaded copy of CA classes, but that classloader
can't see your classloader, which also has CA, but also the fastutil it needs.
Shading CA disambiguates this. (So, put that workaround in your pocket for now;
you should be able to do this with SBT.)
This is what the userClassPathFirst stuff is supposed to resolve though.
You're definitely sure CA is in your app JAR? I ask just because you
mention it was marked provided above, though also in your assembly. Worth
double-checking.
Otherwise, I'm not sure. It almost sounds like the inverse of unresolved
(?) https://issues.apache.org/jira/browse/SPARK-1863 but may surprise me by
being the same issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
