[
https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986490#comment-13986490
]
Sean Owen commented on SPARK-1693:
----------------------------------
I think this is traceable to a case of jar conflict. I am not sure whether the
ultimate cause is signing, but it doesn't matter, since we should simply
resolve the conflict. (But I think something like that may be at play, since
one of the problem dependencies is from Eclipse's jetty, and there is an
Eclipse cert in the manifest at META-INF/ECLIPSEF.RSA...) Anyway.
This is another fun jar hell puzzler, albeit one with a probable solution. The
basic issues is that Jetty brings in the Servlet 3.0 API:
{code}
[INFO] | +- org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:compile
[INFO] | | +-
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
{code}
... but in Hadoop 2.3.0+, so does Hadoop client:
{code}
[INFO] | +- org.apache.hadoop:hadoop-client:jar:2.4.0:compile
...
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.4.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.4.0:compile
[INFO] | | | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
...
[INFO] | | | +- javax.servlet:servlet-api:jar:2.5:compile
{code}
Eclipse is naughty for packaging the same API classes in a different artifact,
rather than just using javax.servlet:servlet-api 3.0. There may be a reason for
that, which is what worries me. In theory Servlet 3.0 is a superset of 2.5, so
Hadoop's client should be happy with the 3.0 API on the classpath.
So solution #1 to try is to exclude javax.servlet:servlet-api from the
hadoop-client dependency. It won't affect earlier versions at all since they
don't have this dependency, and therefore need not even be version-specific.
Hadoop 2.3+ then in theory should happily find Eclipse's Servlet 3.0 API
classes and work as normal. I give that about an 90% chance of being true.
Solution #2 is more theoretically sound. We should really exclude Jetty's
custom copy of Servlet 3.0, and depend on javax.servlet:servlet-api:jar:3.0 as
a runtime dependency. This should transparently override Hadoop 2.3+'s version,
and still work for Hadoop. Messing with Jetty increases the chance of another
snag. Probability of success: 80%
Li would you be able to try either of those ideas?
> Most of the tests throw a java.lang.SecurityException when spark built for
> hadoop 2.3.0 , 2.4.0
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-1693
> URL: https://issues.apache.org/jira/browse/SPARK-1693
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Guoqiang Li
> Assignee: Guoqiang Li
> Attachments: log.txt
>
>
> {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 >
> log.txt{code}
> The log:
> {code}
> UnpersistSuite:
> - unpersist RDD *** FAILED ***
> java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s
> signer information does not match signer information of other classes in the
> same package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
