This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b7cfae0e4d90 [SPARK-57877][CORE] DirectByteBuffer cleaner support Java
26
b7cfae0e4d90 is described below
commit b7cfae0e4d90a458d0780fc43274891ac22d4268
Author: Cheng Pan <[email protected]>
AuthorDate: Tue Jul 21 23:22:38 2026 -0700
[SPARK-57877][CORE] DirectByteBuffer cleaner support Java 26
### What changes were proposed in this pull request?
[JDK-8344332](https://bugs.openjdk.org/browse/JDK-8344332) migrates
`DirectByteBuffer` away from `jdk.internal.ref.Cleaner` to a new
`java.nio.BufferCleaner`, and
[JDK-8361426](https://bugs.openjdk.org/browse/JDK-8361426) removes
`jdk.internal.ref.Cleaner` entirely in JDK 26.
### Why are the changes needed?
Adapt to upstream JDK change.
Close #56939
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
#### Run PlatformUtilSuite with JDK 26
```
JAVA_HOME=/path/of/openjdk-26 build/sbt "unsafe/testOnly *PlatformUtilSuite"
```
before
```
[info] Test run started (JUnit Jupiter)
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#freeingOnHeapMemoryBlockResetsBaseObjectAndOffset()
started
[error] Test
org.apache.spark.unsafe.PlatformUtilSuite.freeingOnHeapMemoryBlockResetsBaseObjectAndOffset
failed: java.lang.ExceptionInInitializerError: null, took 0.022s
[error] at
org.apache.spark.unsafe.memory.HeapMemoryAllocator.allocate(HeapMemoryAllocator.java:73)
[error] at
org.apache.spark.unsafe.PlatformUtilSuite.freeingOnHeapMemoryBlockResetsBaseObjectAndOffset(PlatformUtilSuite.java:79)
[error] Caused by: java.lang.IllegalStateException:
java.lang.ClassNotFoundException: jdk.internal.ref.Cleaner
[error] at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:104)
[error] ... 2 more
[error] Caused by: java.lang.ClassNotFoundException:
jdk.internal.ref.Cleaner
[error] at
jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:502)
[error] at java.lang.Class.forName0(Native Method)
[error] at java.lang.Class.forName(Class.java:478)
[error] at java.lang.Class.forName(Class.java:468)
[error] at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:84)
[error] ... 2 more
...
```
after
```
[info] Test run started (JUnit Jupiter)
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#freeingOnHeapMemoryBlockResetsBaseObjectAndOffset()
started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#overlappingCopyMemory() started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#memoryDebugFillEnabledInTest() started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#offHeapMemoryAllocatorThrowsAssertionErrorOnDoubleFree()
started
[info] Test org.apache.spark.unsafe.PlatformUtilSuite#heapMemoryReuse()
started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#onHeapMemoryAllocatorPoolingReUsesLongArrays()
started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#onHeapMemoryAllocatorThrowsAssertionErrorOnDoubleFree()
started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#freeingOffHeapMemoryBlockResetsOffset()
started
[info] Test
org.apache.spark.unsafe.PlatformUtilSuite#cleanerCreateMethodIsDefined() started
[info] Test run finished: 0 failed, 0 ignored, 9 total, 0.104s
[info] Passed: Total 9, Failed 0, Errors 0, Passed 9
[success] Total time: 9 s, completed Jul 2, 2026, 2:56:53 PM
```
#### Manually verify with JPMS args
```
cat > /tmp/CheckPlatform.java <<EOF
public class CheckPlatform {
public static void main(String[] a) {
int m =
Integer.parseInt(System.getProperty("java.specification.version"));
Throwable err = null; boolean def = false;
try { def =
org.apache.spark.unsafe.Platform.cleanerCreateMethodIsDefined(); }
catch (Throwable t) { err = t; }
String s = "JDK " + m + " | cleanerCreateMethodIsDefined=" + def;
if (err != null) { Throwable c = err.getCause() != null ?
err.getCause() : err;
s += " | init threw " + err.getClass().getSimpleName() + " <- " +
c.getClass().getName() + ": " + c.getMessage(); }
System.out.println(s);
}
}
EOF
```
```
build/sbt 'unsafe/package'
```
```
NIO='--add-opens=java.base/java.nio=ALL-UNNAMED'
REF='--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED'
```
master branch
```
printf '%-46s ' "JDK25 / no add-opens";
"$JAVA25_HOME/bin/java" -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK25 / java.nio + jdk.internal.ref";
"$JAVA25_HOME/bin/java" $NIO $REF -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK26 / no add-opens";
"$JAVA26_HOME/bin/java" -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK26 / java.nio + jdk.internal.ref";
"$JAVA26_HOME/bin/java" $NIO $REF -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
```
```
JDK25 / no add-opens JDK 25 |
cleanerCreateMethodIsDefined=false
JDK25 / java.nio + jdk.internal.ref JDK 25 |
cleanerCreateMethodIsDefined=true
JDK26 / no add-opens JDK 26 |
cleanerCreateMethodIsDefined=false
JDK26 / java.nio + jdk.internal.ref JDK 26 |
cleanerCreateMethodIsDefined=false | init threw ExceptionInInitializerError <-
java.lang.IllegalStateException: java.lang.ClassNotFoundException:
jdk.internal.ref.Cleaner
```
this PR
```
printf '%-46s ' "JDK25 / no add-opens";
"$JAVA25_HOME/bin/java" -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK25 / java.nio only";
"$JAVA25_HOME/bin/java" $NIO -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK25 / java.nio + jdk.internal.ref";
"$JAVA25_HOME/bin/java" $NIO $REF -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK26 / no add-opens";
"$JAVA26_HOME/bin/java" -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
printf '%-46s ' "JDK26 / java.nio only";
"$JAVA26_HOME/bin/java" $NIO -cp
common/unsafe/target/scala-2.13/spark-unsafe_2.13-5.0.0-SNAPSHOT.jar
/tmp/CheckPlatform.java 2>/dev/null
```
```
JDK25 / no add-opens JDK 25 |
cleanerCreateMethodIsDefined=false
JDK25 / java.nio only JDK 25 |
cleanerCreateMethodIsDefined=false
JDK25 / java.nio + jdk.internal.ref JDK 25 |
cleanerCreateMethodIsDefined=true
JDK26 / no add-opens JDK 26 |
cleanerCreateMethodIsDefined=false
JDK26 / java.nio only JDK 26 |
cleanerCreateMethodIsDefined=true
```
### Was this patch authored or co-authored using generative AI tooling?
Yes, GLM 5.2 is used to construct manual test cases.
Closes #56954 from pan3793/SPARK-57877.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 751f29f12933ef3626518e8ed7a13a225c37d326)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../java/org/apache/spark/unsafe/Platform.java | 28 +++++++++++++++-------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
index 9bb69ec2471b..efd9cc70d01c 100644
--- a/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
+++ b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
@@ -81,16 +81,26 @@ public final class Platform {
// no point continuing if the above failed:
if (DBB_CONSTRUCTOR != null && DBB_CLEANER_FIELD != null) {
- Class<?> cleanerClass = Class.forName("jdk.internal.ref.Cleaner");
- Method createMethod = cleanerClass.getMethod("create", Object.class,
Runnable.class);
- // Accessing jdk.internal.ref.Cleaner should actually fail by default
in JDK 9+,
- // unfortunately, unless the user has allowed access with something
like
- // --add-opens java.base/jdk.internal.ref=ALL-UNNAMED If not, we
can't use the Cleaner
- // hack below. It doesn't break, just means the user might run into
the default JVM limit
- // on off-heap memory and increase it or set the flag above. This
tests whether it's
- // available:
+ // JDK-8344332 (JDK 26) migrate DirectByteBuffer away from
jdk.internal.ref.Cleaner
+ Class<?> cleanerClass = (majorVersion < 26) ?
+ Class.forName("jdk.internal.ref.Cleaner") :
+ Class.forName("java.nio.BufferCleaner");
+ Method createMethod = (majorVersion < 26) ?
+ cleanerClass.getMethod("create", Object.class, Runnable.class) :
+ cleanerClass.getMethod("register", Object.class, Runnable.class);
+ // Accessing jdk.internal.ref.Cleaner (JDK 9~25) /
java.nio.BufferCleaner (JDK 26+)
+ // should actually fail by default, unless the user has allowed access
with something like
+ // --add-opens java.base/jdk.internal.ref=ALL-UNNAMED (JDK 9~25)
+ // --add-opens java.base/java.nio=ALL-UNNAMED (JDK 26+)
+ // If not, we can't use the Cleaner hack below. It doesn't break, just
means the user might
+ // run into the default JVM limit on off-heap memory and increase it
or set the flag above.
+ // This tests whether it's available:
try {
- createMethod.invoke(null, null, null);
+ if (createMethod.trySetAccessible()) {
+ createMethod.invoke(null, new Object(), (Runnable) () -> {});
+ } else {
+ createMethod = null;
+ }
} catch (IllegalAccessException e) {
// Don't throw an exception, but can't log here?
createMethod = null;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
