LuciferYang opened a new pull request #34368:
URL: https://github.com/apache/spark/pull/34368
### What changes were proposed in this pull request?
There are 9 UTs failed in `repl` run with Java 17, the all failed tests have
similar error stack as follows:
```
java.lang.IllegalAccessException: Can not set final $iw field
$Lambda$2879/0x000000080188b928.arg$1 to $iw
at
java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwFinalFieldIllegalAccessException(UnsafeFieldAccessorImpl.java:76)
at
java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwFinalFieldIllegalAccessException(UnsafeFieldAccessorImpl.java:80)
at
java.base/jdk.internal.reflect.UnsafeQualifiedObjectFieldAccessorImpl.set(UnsafeQualifiedObjectFieldAccessorImpl.java:79)
at java.base/java.lang.reflect.Field.set(Field.java:799)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:398)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2490)
at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:414)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
at org.apache.spark.rdd.RDD.map(RDD.scala:413)
... 95 elided
```
We know from error stack that we can't reset a `read-only` field `arg$1` to
`$iw` through the reflection api when use Java 17, the `read-only` field is
defined as `private final field (trustedFinal=true)` .
So this pr try to remove the `final` modifier before `outerField.set(func,
clonedOuterThis)` in `ClosureCleaner#clean` method and reset the modifier
value after `outerField.set(func, clonedOuterThis)` when use Java 17.
At the same time, in order to use the API in `java.lang.reflect` mod,
`--add-opens=java.base/java.lang.reflect=ALL-UNNAMED` is added to `pom.xml`,
`SparkBuild.scala` and `JavaModuleOptions.java`
### Why are the changes needed?
Pass UT with JDK 17
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test use Java 17
```
mvn clean install -pl repl
```
**Before**
```
Run completed in 30 seconds, 826 milliseconds.
Total number of tests run: 42
Suites: completed 6, aborted 0
Tests: succeeded 33, failed 9, canceled 0, ignored 0, pending 0
*** 9 TESTS FAILED ***
```
**After**
```
Run completed in 45 seconds, 86 milliseconds.
Total number of tests run: 44
Suites: completed 7, aborted 0
Tests: succeeded 44, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
