Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9615#issuecomment-157227910
Ok, so I figured out why you were seeing the exceptions with
`--driver-class-path`.
The old version of jersey hooks up to the JAX-RS API using
`META-INF/services` files; the new version doesn't. So, when you have both the
new jersey and the old jersey in the classpath, the JAX-RS code will load
classes from the old jersey, because that's what the `META-INF` entries tell it
to do.
The old jersey implementation of the `Response` class does not have the
`getStatusInfo` method, so you see that error.
This also means that plain shading will not work. You would have to modify
all the services files too; not sure whether the plugins does that, but I don't
think so. And that's assuming that shading JAX-RS annotations doesn't break
other things.
One thing I tried and seems to have worked in a quick test is to not
package the old jersey service files in the assembly. I just added this to the
sbt build:
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 67724c4..e8317b0 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -445,6 +445,7 @@ object Assembly {
case m if m.toLowerCase.endsWith("manifest.mf") =>
MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") =>
MergeStrategy.discard
case "log4j.properties" =>
MergeStrategy.discard
+ case m if m.toLowerCase.startsWith("meta-inf/services/javax.ws")
=> MergeStrategy.discard
case m if m.toLowerCase.startsWith("meta-inf/services/") =>
MergeStrategy.filterDistinctLines
case "reference.conf" =>
MergeStrategy.concat
case _ =>
MergeStrategy.first
When I added the new jersey to the class path, the new classes were loaded
(and worked), and the Spark UI was still able to serve things under `/api`.
Without the new classes, `/api` also works. So it seems there might be a way
out even without shading.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
