[
https://issues.apache.org/jira/browse/SPARK-11081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042048#comment-15042048
]
Andrew Ash commented on SPARK-11081:
------------------------------------
Agreed, Spark 2.0 is the main opportunity to bump as many dependency versions
as possible. Some of these constraints are imposed on Spark by its transitive
dependencies, not the direct ones though. But I totally support doing a
dependency inspection during the 2.0 phase.
> Make spark-core pull in Jersey and javax.ws.rs dependencies separately for
> easier overriding
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-11081
> URL: https://issues.apache.org/jira/browse/SPARK-11081
> Project: Spark
> Issue Type: Improvement
> Components: Build, Spark Core
> Reporter: Mingyu Kim
>
> As seen from this thread
> (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=vrx-d2...@mail.gmail.com%3E),
> Spark is incompatible with Jersey 2 especially when Spark is embedded in an
> application running with Jersey.
> There was an in-depth discussion on options for shading and making it easier
> for users to be able to use Jersey 2 with Spark applications:
> https://github.com/apache/spark/pull/9615
> To recap the discussion, Jersey 1 has two issues:
> 1. It has classes listed in META-INF/services/ files that would be loaded
> even if Jersey 2 was being loaded on the classpath in a higher precedence.
> This means that Jersey 2 would attempt to use Jersey 1 implementations in
> some places regardless of user attempts to override the dependency with
> things like userClassPathFirst.
> 2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to
> exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is
> compatible with.
> Also discussed was the fact that plain old shading doesn't work here, since
> you would need to shade lines in META-INF/services as well, not just classes.
> Not to mention that shading JAX-RS annotations is tricky as well.
> To recap the discussion as what needs to happen Spark-side, we need to:
> 1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is
> the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually
> shade/namespace the classes that way, just the artifact name)
> 2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact,
> say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files
> live in this artifact as well)
> 3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
> 4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out
> these files aren't actually necessary for Jersey 1 to function properly in
> general (we need to test this more however)
> Now a user that wants to depend on Jersey 2, and is depending on Spark maven
> artifacts, would do the following in their application
> 1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs
> dependencies
> 2. In my application's dependencies, exclude org.spark-project.javax.ws.rs
> from spark-core. We keep org.spark-project.jersey because spark-core needs
> it, but it will use the javax.ws.rs classes that my application is providing.
> 3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new
> javax.ws.rs jars to the executors
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
