This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 16295a33822 [SPARK-44297][CORE][TESTS] Make
`ClassLoaderIsolationSuite` test pass with Scala 2.13
16295a33822 is described below
commit 16295a338220cc66fea5d91bcbd4213df0f2d4bd
Author: yangjie01 <[email protected]>
AuthorDate: Tue Jul 4 22:37:04 2023 +0800
[SPARK-44297][CORE][TESTS] Make `ClassLoaderIsolationSuite` test pass with
Scala 2.13
### What changes were proposed in this pull request?
The main change of this pr as follows:
1. rename `TestHelloV2.jar` and `TestHelloV3.jar` added in
https://github.com/apache/spark/pull/41789 to `TestHelloV2_2.12.jar` and
`TestHelloV3_2.12.jar`
2. Add corresponding `TestHelloV2_2.13.jar` and `TestHelloV3_2.13.jar`
which compiled with Scala 2.13
3. Make `ClassLoaderIsolationSuite` use the correct jar in testing
### Why are the changes needed?
Make `ClassLoaderIsolationSuite` test pass with Scala 2.13.
The Scala 2.13 daily test failed after
https://github.com/apache/spark/pull/41789
- https://github.com/apache/spark/actions/runs/5447771717/jobs/9910185372
```
[info] - Executor classloader isolation with JobArtifactSet *** FAILED ***
(83 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage
0.0 (TID 0) (localhost executor driver): java.lang.NoClassDefFoundError:
scala/Serializable
[info] at java.lang.ClassLoader.defineClass1(Native Method)
[info] at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
[info] at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
[info] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
[info] at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
[info] at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
[info] at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
[info] at java.security.AccessController.doPrivileged(Native Method)
[info] at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
[info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[info] at java.lang.Class.forName0(Native Method)
[info] at java.lang.Class.forName(Class.java:348)
[info] at
org.apache.spark.util.SparkClassUtils.classForName(SparkClassUtils.scala:35)
[info] at
org.apache.spark.util.SparkClassUtils.classForName$(SparkClassUtils.scala:30)
[info] at org.apache.spark.util.Utils$.classForName(Utils.scala:94)
[info] at
org.apache.spark.executor.ClassLoaderIsolationSuite.$anonfun$new$3(ClassLoaderIsolationSuite.scala:53)
[info] at
scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
[info] at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
[info] at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
[info] at
org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
[info] at org.apache.spark.rdd.RDD.$anonfun$foreach$2(RDD.scala:1028)
[info] at
org.apache.spark.rdd.RDD.$anonfun$foreach$2$adapted(RDD.scala:1028)
[info] at
org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2406)
[info] at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
[info] at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
[info] at org.apache.spark.scheduler.Task.run(Task.scala:141)
[info] at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:593)
[info] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1478)
[info] at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:596)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] at java.lang.Thread.run(Thread.java:750)
[info] Caused by: java.lang.ClassNotFoundException: scala.Serializable
[info] at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
[info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[info] ... 33 more
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual check:
**Scala 2.12**
```
build/sbt "core/testOnly *ClassLoaderIsolationSuite"
```
```
[info] ClassLoaderIsolationSuite:
[info] - Executor classloader isolation with JobArtifactSet (1 second, 394
milliseconds)
[info] Run completed in 2 seconds, 437 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
**Scala 2.13**
```
dev/change-scala-version.sh 2.13
build/sbt "core/testOnly *ClassLoaderIsolationSuite" -Pscala-2.13
```
```
[info] ClassLoaderIsolationSuite:
[info] - Executor classloader isolation with JobArtifactSet (1 second, 355
milliseconds)
[info] Run completed in 2 seconds, 264 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
Closes #41852 from LuciferYang/SPARK-44297.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.../resources/{TestHelloV2.jar => TestHelloV2_2.12.jar} | Bin
core/src/test/resources/TestHelloV2_2.13.jar | Bin 0 -> 4118 bytes
.../resources/{TestHelloV3.jar => TestHelloV3_2.12.jar} | Bin
core/src/test/resources/TestHelloV3_2.13.jar | Bin 0 -> 3827 bytes
.../spark/executor/ClassLoaderIsolationSuite.scala | 14 ++++++++++++--
5 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/core/src/test/resources/TestHelloV2.jar
b/core/src/test/resources/TestHelloV2_2.12.jar
similarity index 100%
rename from core/src/test/resources/TestHelloV2.jar
rename to core/src/test/resources/TestHelloV2_2.12.jar
diff --git a/core/src/test/resources/TestHelloV2_2.13.jar
b/core/src/test/resources/TestHelloV2_2.13.jar
new file mode 100644
index 00000000000..6dee8fcd9c9
Binary files /dev/null and b/core/src/test/resources/TestHelloV2_2.13.jar differ
diff --git a/core/src/test/resources/TestHelloV3.jar
b/core/src/test/resources/TestHelloV3_2.12.jar
similarity index 100%
rename from core/src/test/resources/TestHelloV3.jar
rename to core/src/test/resources/TestHelloV3_2.12.jar
diff --git a/core/src/test/resources/TestHelloV3_2.13.jar
b/core/src/test/resources/TestHelloV3_2.13.jar
new file mode 100644
index 00000000000..0c292e7d81a
Binary files /dev/null and b/core/src/test/resources/TestHelloV3_2.13.jar differ
diff --git
a/core/src/test/scala/org/apache/spark/executor/ClassLoaderIsolationSuite.scala
b/core/src/test/scala/org/apache/spark/executor/ClassLoaderIsolationSuite.scala
index 33c1baccd72..72ee0e96fd0 100644
---
a/core/src/test/scala/org/apache/spark/executor/ClassLoaderIsolationSuite.scala
+++
b/core/src/test/scala/org/apache/spark/executor/ClassLoaderIsolationSuite.scala
@@ -17,21 +17,31 @@
package org.apache.spark.executor
+import scala.util.Properties
+
import org.apache.spark.{JobArtifactSet, LocalSparkContext, SparkConf,
SparkContext, SparkFunSuite}
import org.apache.spark.util.Utils
class ClassLoaderIsolationSuite extends SparkFunSuite with LocalSparkContext {
+
+ private val scalaVersion = Properties.versionNumberString
+ .split("\\.")
+ .take(2)
+ .mkString(".")
+
val jar1 =
Thread.currentThread().getContextClassLoader.getResource("TestUDTF.jar").toString
// package com.example
// object Hello { def test(): Int = 2 }
// case class Hello(x: Int, y: Int)
- val jar2 =
Thread.currentThread().getContextClassLoader.getResource("TestHelloV2.jar").toString
+ val jar2 = Thread.currentThread().getContextClassLoader
+ .getResource(s"TestHelloV2_$scalaVersion.jar").toString
// package com.example
// object Hello { def test(): Int = 3 }
// case class Hello(x: String)
- val jar3 =
Thread.currentThread().getContextClassLoader.getResource("TestHelloV3.jar").toString
+ val jar3 = Thread.currentThread().getContextClassLoader
+ .getResource(s"TestHelloV3_$scalaVersion.jar").toString
test("Executor classloader isolation with JobArtifactSet") {
sc = new SparkContext(new
SparkConf().setAppName("test").setMaster("local"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
