Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2984#issuecomment-60849659
I wanted to comment on the bug after I investigated some things, but since
you filed a PR, let me comment here...
I don't think this is a good long term solution (yeah, I know, ironic from
the guy who worked on the Guava shading thing :-)). I think the right (or,
let's say, better) way to do this is to actually make
`spark.files.userClassPathFirst` work.
The goal of that setting is to make the user's classpath run in a separate
class loader from the Spark classes. So you'd package your app with your jetty,
and turn on that setting; your app's classes would come from this separate
class loader, and Spark classes would come from the system one. Your app would
see your Jetty, Spark would see its own. All without shading. (There are
caveats, but it should mostly work. Hadoop/MR has something similar to this,
too.)
The problem is that setting seems to be sort of broken. It's definitely
broken in any client mode, since SparkSubmit.scala doesn't seem to be doing the
right thing. And it's definitely broken in Yarn, since it's completely ignored
there...
In any case, long term, I think the class loader isolation thing is a
better way to go, otherwise we'll have to start changing the Spark build every
time someone wants to use a different version of some library...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
