This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a81fc2a65014 [SPARK-55670][BUILD] Add `-Dio.netty.noUnsafe=false` to
enable Java 25 support
a81fc2a65014 is described below
commit a81fc2a65014b45940ced461e06f8b3e19f36b2f
Author: Cheng Pan <[email protected]>
AuthorDate: Wed Feb 25 13:31:14 2026 +0800
[SPARK-55670][BUILD] Add `-Dio.netty.noUnsafe=false` to enable Java 25
support
### What changes were proposed in this pull request?
Arrow requires `-Dio.netty.noUnsafe=false` for Java 24+, see
https://github.com/apache/arrow-java/issues/728
### Why are the changes needed?
To enable Java 25 support, especially for PySpark, which heavily uses Arrow
for JVM/Python data exchange.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually verified with JDK 25 (we don't have JDK 25 CI yet), take one Arrow
related UT as example.
```
$ export JAVA_HOME=/path/of/openjdk-25
$ build/sbt "sql/testOnly *ArrowConvertersSuite"
```
Before
```
[info] Caused by: java.lang.ExceptionInInitializerError: Exception
java.lang.ExceptionInInitializerError [in thread "Executor task launch worker
for task 0.0 in stage 0.0 (TID 0)"
[info] at
org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
[info] at java.base/java.lang.Class.forName0(Native Method)
[info] at java.base/java.lang.Class.forName(Class.java:467)
[info] at java.base/java.lang.Class.forName(Class.java:458)
[info] at
org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:105)
[info] at
org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:92)
[info] at
org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:826)
[info] at
org.apache.arrow.memory.ImmutableConfig.access$001(ImmutableConfig.java:20)
[info] at
org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:80)
[info] at
org.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:43)
[info] at
org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:487)
[info] at
org.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:72)
[info] at
org.apache.spark.sql.util.ArrowUtils$.<clinit>(ArrowUtils.scala:36)
...
[info] Run completed in 6 seconds, 326 milliseconds.
[info] Total number of tests run: 26
[info] Suites: completed 0, aborted 1
[info] Tests: succeeded 0, failed 26, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
[info] *** 26 TESTS FAILED ***
[error] Error during tests:
[error] org.apache.spark.sql.execution.arrow.ArrowConvertersSuite
[error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 203 s (0:03:23.0), completed Feb 25, 2026, 10:39:52 AM
```
After
```
[info] Run completed in 7 seconds, 17 milliseconds.
[info] Total number of tests run: 43
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 43, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 29 s, completed Feb 25, 2026, 10:41:46 AM
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54470 from pan3793/SPARK-55670.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java | 1 +
pom.xml | 1 +
project/SparkBuild.scala | 1 +
sql/connect/bin/spark-connect-scala-client | 1 +
4 files changed, 4 insertions(+)
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
b/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
index b42c7906fe18..49c24bc88783 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
@@ -47,6 +47,7 @@ public class JavaModuleOptions {
"-Dio.netty.tryReflectionSetAccessible=true",
"-Dio.netty.allocator.type=pooled",
"-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE",
+ "-Dio.netty.noUnsafe=false",
"--enable-native-access=ALL-UNNAMED"};
/**
diff --git a/pom.xml b/pom.xml
index ff73a3041ebe..b73e44079663 100644
--- a/pom.xml
+++ b/pom.xml
@@ -335,6 +335,7 @@
-Dio.netty.tryReflectionSetAccessible=true
-Dio.netty.allocator.type=pooled
-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE
+ -Dio.netty.noUnsafe=false
--enable-native-access=ALL-UNNAMED
</extraJavaTestArgs>
<mariadb.java.client.version>3.5.7</mariadb.java.client.version>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index ebd553e400e2..674d97c16086 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1876,6 +1876,7 @@ object TestSettings {
"-Dio.netty.tryReflectionSetAccessible=true",
"-Dio.netty.allocator.type=pooled",
"-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE",
+ "-Dio.netty.noUnsafe=false",
"--enable-native-access=ALL-UNNAMED").mkString(" ")
s"-Xmx$heapSize -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize
-XX:ReservedCodeCacheSize=128m -Dfile.encoding=UTF-8 $extraTestJavaArgs"
.split(" ").toSeq
diff --git a/sql/connect/bin/spark-connect-scala-client
b/sql/connect/bin/spark-connect-scala-client
index 7f90353cc39b..085f7e92f3c2 100755
--- a/sql/connect/bin/spark-connect-scala-client
+++ b/sql/connect/bin/spark-connect-scala-client
@@ -73,6 +73,7 @@ JVM_ARGS="-XX:+IgnoreUnrecognizedVMOptions \
-Dio.netty.tryReflectionSetAccessible=true \
-Dio.netty.allocator.type=pooled \
-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE \
+ -Dio.netty.noUnsafe=false \
--enable-native-access=ALL-UNNAMED \
$SCJVM_ARGS"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
