[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893074#comment-15893074 ] Andrew Otto commented on SPARK-1693: We just upgraded to CDH 5.10, which has Spark 1.6.0, Hadoop 2.6.0, Hive 1.1.0, and Oozie 4.1.0. We are having trouble running Spark jobs that use HiveContext from Oozie. They run perfectly fine from the CLI with spark-submit, just not in Oozie. We aren't certain that HiveContext is related, but we can reproduce regularly with a job that uses HiveContext. Anyway, I post this here, because the error we are getting is the same that started this issue: {code}class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package{code} I've noticed that the Oozie sharelib includes javax.servlet-3.0.0.v201112011016.jar. I also see that spark-assembly.jar includes a javax.servlet.FilterRegistration class, although its hard for me to tell which version. The jetty pom.xml files in spark-assembly.jar seem to say {{javax.servlet.*;version="2.6.0"}}, but I'm a little green on how all these dependencies get resolved. I don't see any javax.servlet .jars in any of /usr/lib/hadoop* (where CDH installs hadoop jars). Help! :) If this is not related to this issue, I'll open a new one. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089945#comment-15089945 ] Sean Owen commented on SPARK-1693: -- As I recall this happens because the official JavaEE servlet API jar includes MANIFEST.MF entries with a signature. If these classes are repackaged in an assembly jar with these same MANIFEST.MF entries, you get this failure. There's a lot of discussion above, but one quick fix is to manage to not include the signatures from anywhere. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089936#comment-15089936 ] Owen O'Malley commented on SPARK-1693: -- Can you explain what the problem is and how to fix it? We are hitting the same problem on the hive on spark work. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728669#comment-14728669 ] Hudson commented on SPARK-1693: --- SUCCESS: Integrated in gora-trunk #1610 (See [https://builds.apache.org/job/gora-trunk/1610/]) servlet dependency is excluded from spark, not ant. Without excluding it throws a java security exception. Motivation for it: https://issues.apache.org/jira/browse/SPARK-1693 and (furkankamaci: rev ad4f5601ebd52e182bf79ba4032dcb85d918b925) * gora-core/pom.xml * pom.xml > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562555#comment-14562555 ] Michael Schmeißer commented on SPARK-1693: -- I think the fix version of this issue is incorrect, I could reproduce the problem with spark-core 1.1.0-cdh5.2.1. With version 1.3.0-cdh5.4.2, the same code works fine in local execution. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988618#comment-13988618 ] Guoqiang Li commented on SPARK-1693: [~srowen] You're right,preferably only one version exists. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988615#comment-13988615 ] Sean Owen commented on SPARK-1693: -- [~tgraves] I see it in tests. It sounds like it does not affect simple, normal operation, which is good. It would still be good to fix up, for propiety's sake and because of the tests. [~gq] the problem is exactly that several copies of the same class are merged, right? > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988534#comment-13988534 ] Guoqiang Li commented on SPARK-1693: [~tgraves] You're right . Different versions of the Servlet APIs are forced to merge into spark-assembly.*hadoop.*.jar,each .class file will only have a presence > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988212#comment-13988212 ] Thomas Graves commented on SPARK-1693: -- Is this only seen in the tests? When I run on a real hadoop2.4 cluster with spark built for hadoop 2.4 I don't see any exceptions. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987454#comment-13987454 ] Sean Owen commented on SPARK-1693: -- [~pwendell] The Hadoop artifacts want version 2.5, as you see. The Servlet 3.0.x *APIs* should be a superset of 2.5. In theory clients implementing or using 2.5 will be fine with it, which means there's an 80% chance it's actually true. Hence I think a solution is to enforce use of 3.0.1, and exclude all other copies of the servlet API. Normally that's as simple as declaring a runtime dependency on the latest version of the artifact, which will override any transitive dependencies. The problem here is exactly because that's not true; Eclipse has its own artifact for the servlet 3.0 APIs. And [~witgo], as you noted, the artifact name actually changed from 2.5 to 3.0.1! So depending on it will *not* remove Hadoop's dependency. I think one more change is needed, to exclude Hadoop's dependency on the old javax.servlet:servlet-api artifact. Basically, "mvn ... dependency:tree | grep javax.servlet" should only show the new 3.0.1 dependency. This should hopefully solve the client-side, unit test failures at least, which is necessary. [~witgo] I assume you have found it makes the tests pass? I do not think it will affect compatibility with a cluster, as this isn't changing or affecting how the client talks to a cluster. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987323#comment-13987323 ] Guoqiang Li commented on SPARK-1693: Now, the main problem is to ensure that hadoop 2.3, 2.4 can work in this version.But I don't have Hadoop 2.3, 2.4 clusters. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li >Priority: Blocker > Fix For: 1.0.0 > > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987316#comment-13987316 ] Patrick Wendell commented on SPARK-1693: I'm confused about something [~sowen]: If Hadoop 2.3/2.4 wants version 3 of the servlet-api dependency, why does the Hadoop build declare version 2.5 as a dependency in their POM's? http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.3.0/hadoop-project-2.3.0.pom http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.4.0/hadoop-project-2.4.0.pom Also - is there not a risk of bumping this version to 3.X when users are building against older hadoop versions? I noticed the older hadoop versions depend 2.X versions of this... will it break them if 2.X and 3.X are present at runtime? [~sowen] If you could look over this PR it would be great. It's really scary merging a big change like this to the build at the 11th hour. So the more eyes the better. > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986656#comment-13986656 ] Guoqiang Li commented on SPARK-1693: Thanks Sean Owen . Code has been submitted to [PR 590|https://github.com/apache/spark/pull/590] > Dependent on multiple versions of servlet-api jars lead to throw an > SecurityException when Spark built for hadoop 2.3.0 , 2.4.0 > > > Key: SPARK-1693 > URL: https://issues.apache.org/jira/browse/SPARK-1693 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Guoqiang Li >Assignee: Guoqiang Li > Attachments: log.txt > > > {code}mvn test -Pyarn -Dhadoop.version=2.4.0 -Dyarn.version=2.4.0 > > log.txt{code} > The log: > {code} > UnpersistSuite: > - unpersist RDD *** FAILED *** > java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s > signer information does not match signer information of other classes in the > same package > at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) > at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) > at java.lang.ClassLoader.defineClass(ClassLoader.java:794) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)