[
https://issues.apache.org/jira/browse/SPARK-32954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-32954:
-----------------------------
Description:
When I tried to verify that the resource-managers yarn module passed all UTs in
Scala 2.13 , I found that there are a classpath dependent badcase blocking the
test because there are more than one `servlet-api` dependency in spark now:
* one is `javax.servlet:javax.servlet-api:3.10:compile` we config in
core/pom.xml,
* the other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test` cascaded by
`org.glassfish.jersey.test-framework.providers`, we can use `mvn
dependency:tree` to check it .
So when we execute resource-managers yarn module test use
mvn clean test -pl resource-managers/yarn -Pyarn
or
mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13
if the position of `javax.servlet-api` in the in classpath is before
`jakarta.servlet-api`, there are some cases failed in YarnClusterSuite,
YarnShuffleIntegrationSuite and YarnShuffleAuthSuite.
The failed reason as follow:
{code:java}
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Exception in
thread "main" java.lang.ExceptionInInitializerError
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:117)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:104)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.metrics.sink.MetricsServlet.getHandlers(MetricsServlet.scala:53)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.metrics.MetricsSystem.$anonfun$getServletHandlers$2(MetricsSystem.scala:93)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
scala.Option.map(Option.scala:230)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:93)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.SparkContext.<init>(SparkContext.scala:582)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.yarn.YarnClusterDriver$.main(YarnClusterSuite.scala:413)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.yarn.YarnClusterDriver.main(YarnClusterSuite.scala)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
java.lang.reflect.Method.invoke(Method.java:498)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:934)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1013)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1022)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Caused by:
java.lang.SecurityException: class "javax.servlet.http.HttpSessionIdListener"'s
signer information does not match signer information of other classes in the
same package
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
java.lang.ClassLoader.defineClass(ClassLoader.java:761)
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
20/09/18 19:14:07.487 launcher-proc-1 INFO YarnClusterDriver: at
java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
20/09/18 19:14:07.487 launcher-proc-1 INFO YarnClusterDriver: at
java.net.URLClassLoader.access$100(URLClassLoader.java:73)
20/09/18 19:14:07.487 launcher-proc-1 INFO YarnClusterDriver: at
java.net.URLClassLoader$1.run(URLClassLoader.java:368)
{code}
was:
When I tried to verify that the resource-managers yarn module passed all UTs in
Scala 2.13 , I found that there are some problems blocking the test:
# resource-managers/yarn module is not explicit dependency on sql/hive module
but some case need HiveConf like `run Python application in yarn-client mode`
in YarnClusterSuite, so we can't test resource-managers/yarn module
independently as follow:
## mvn clean install -DskipTests -pl resource-managers/yarn -Pyarn
-Pscala-2.13 -am
## mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13
# There are more than one `servlet-api` dependency in spark now, one is
`javax.servlet:javax.servlet-api:3.10:compile` we config in core/pom.xml, the
other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test` cascaded by
`org.glassfish.jersey.test-framework.providers`, we can use `mvn
dependency:tree` to check it .So when we execute resource-managers yarn module
test independently, if the position of
`javax.servlet:javax.servlet-api:3.10:compile` in the in classpath is before
`jakarta.servlet:jakarta.servlet-api:4.0.3:test`, there are some cases failed
in YarnClusterSuite, YarnShuffleIntegrationSuite and YarnShuffleAuthSuite
> Add jakarta.servlet-api test dependency to yarn module to avoid classpath
> badcase of UTs
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-32954
> URL: https://issues.apache.org/jira/browse/SPARK-32954
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.1.0
> Reporter: Yang Jie
> Priority: Minor
>
> When I tried to verify that the resource-managers yarn module passed all UTs
> in Scala 2.13 , I found that there are a classpath dependent badcase blocking
> the test because there are more than one `servlet-api` dependency in spark
> now:
> * one is `javax.servlet:javax.servlet-api:3.10:compile` we config in
> core/pom.xml,
> * the other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test` cascaded by
> `org.glassfish.jersey.test-framework.providers`, we can use `mvn
> dependency:tree` to check it .
> So when we execute resource-managers yarn module test use
> mvn clean test -pl resource-managers/yarn -Pyarn
> or
> mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13
>
> if the position of `javax.servlet-api` in the in classpath is before
> `jakarta.servlet-api`, there are some cases failed in YarnClusterSuite,
> YarnShuffleIntegrationSuite and YarnShuffleAuthSuite.
>
> The failed reason as follow:
>
> {code:java}
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Exception in
> thread "main" java.lang.ExceptionInInitializerError
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:117)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:104)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.metrics.sink.MetricsServlet.getHandlers(MetricsServlet.scala:53)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.metrics.MetricsSystem.$anonfun$getServletHandlers$2(MetricsSystem.scala:93)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> scala.Option.map(Option.scala:230)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:93)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:582)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.yarn.YarnClusterDriver$.main(YarnClusterSuite.scala:413)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.yarn.YarnClusterDriver.main(YarnClusterSuite.scala)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> java.lang.reflect.Method.invoke(Method.java:498)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:934)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1013)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1022)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Caused by:
> java.lang.SecurityException: class
> "javax.servlet.http.HttpSessionIdListener"'s signer information does not
> match signer information of other classes in the same package
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> java.lang.ClassLoader.defineClass(ClassLoader.java:761)
> 20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> 20/09/18 19:14:07.487 launcher-proc-1 INFO YarnClusterDriver: at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> 20/09/18 19:14:07.487 launcher-proc-1 INFO YarnClusterDriver: at
> java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> 20/09/18 19:14:07.487 launcher-proc-1 INFO YarnClusterDriver: at
> java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
