LuciferYang opened a new pull request #34381:
URL: https://github.com/apache/spark/pull/34381
### What changes were proposed in this pull request?
There are 13 UTs failed when test with Java 17 due to
```
ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at
org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at
org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514)
at
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928)
at
org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:87)
at
scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
at scala.concurrent.Promise.tryFailure(Promise.scala:112)
at scala.concurrent.Promise.tryFailure$(Promise.scala:112)
at
scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761)
Caused by: java.lang.IllegalAccessError: class
org.apache.spark.storage.StorageUtils$ (in unnamed module @0x2a48d10f) cannot
access class sun.nio.ch.DirectBuffer (in module java.base) because module
java.base does not export sun.nio.ch to unnamed module @0x2a48d10f
at
org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
at
org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
at
org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
at
org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at
org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:278)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:463)
at
org.apache.spark.deploy.yarn.YarnExternalShuffleDriver$.main(YarnShuffleIntegrationSuite.scala:130)
at
org.apache.spark.deploy.yarn.YarnExternalShuffleDriver.main(YarnShuffleIntegrationSuite.scala)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
```
So this pr add `JavaModuleOptions.defaultModuleOptions()` by default for AM
startup, and it will not affect Java 8 and Java 11 due to
`JavaModuleOptions.defaultModuleOptions()` includes
`-XX:+IgnoreUnrecognizedVMOptions`.
### Why are the changes needed?
Pass UT with JDK 17
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test use Java17 build + Java17 run (both openjdk version "17"
2021-09-14 LTS)
Use Java 17 to run the following two commands
```
build/mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn
build/mvn test -pl resource-managers/yarn -Pyarn
```
**Before**
```
Run completed in 5 minutes, 26 seconds.
Total number of tests run: 144
Suites: completed 19, aborted 0
Tests: succeeded 131, failed 13, canceled 1, ignored 0, pending 0
*** 13 TESTS FAILED ***
```
**After**
```
Run completed in 5 minutes, 52 seconds.
Total number of tests run: 144
Suites: completed 19, aborted 0
Tests: succeeded 144, failed 0, canceled 1, ignored 0, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
