andrey-borisov-patrianna opened a new issue, #56939:
URL: https://github.com/apache/spark/issues/56939
---
## Spark 4.2.0-preview5 fails at SparkContext init on JDK 26:
ClassNotFoundException: jdk.internal.ref.Cleaner
### Description
`org.apache.spark.unsafe.Platform` fails to initialize on **JDK 26**
because its
static initializer attempts to load `jdk.internal.ref.Cleaner` by name,
which no
longer exists / is no longer accessible in JDK 26. The failure surfaces as
an
`ExceptionInInitializerError` the first time
`SparkSession.builder().getOrCreate()`
creates a `SparkContext`, making Spark completely unusable on JDK 26.
The root cause is in `Platform.<clinit>` (Platform.java:84 / :104), which
resolves a
`Cleaner` class via `Class.forName("jdk.internal.ref.Cleaner")` and
rethrows the
`ClassNotFoundException` wrapped in an `IllegalStateException`.
### Affected version
- **Spark:** 4.2.0-preview5
- **Scala:** 2.13.18
- **JDK:** OpenJDK 26.0.1 (Homebrew build 26.0.1)
- **OS:** macOS (Darwin 25.5.0)
### Steps to reproduce
1. Build/run with JDK 26.
2. Create a local `SparkSession`:
```java
SparkSession.builder()
.master("local[*]")
.getOrCreate();
Actual behavior
java.lang.ExceptionInInitializerError
at
org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:47)
at
org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:261)
at
org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:251)
at
org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:271)
at
scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.Option.getOrElse(Option.scala:201)
at
org.apache.spark.memory.MemoryManager.<init>(MemoryManager.scala:271)
at
org.apache.spark.memory.UnifiedMemoryManager.<init>(UnifiedMemoryManager.scala:67)
at
org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:456)
at
org.apache.spark.SparkEnv.initializeMemoryManager(SparkEnv.scala:261)
at
org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:456)
at org.apache.spark.SparkEnv.initializeMemoryManager(SparkEnv.scala:261)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:597)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:3075)
at
org.apache.spark.sql.classic.SparkSession$Builder.$anonfun$build$2(SparkSession.scala:1045)
at scala.Option.getOrElse(Option.scala:201)
at
org.apache.spark.sql.classic.SparkSession$Builder.build(SparkSession.scala:1036)
at
org.apache.spark.sql.classic.SparkSession$Builder.getOrCreate(SparkSession.scala:1065)
at
org.apache.spark.sql.classic.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:929)
Caused by: java.lang.IllegalStateException:
java.lang.ClassNotFoundException: jdk.internal.ref.Cleaner
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:104)
... 19 more
Caused by: java.lang.ClassNotFoundException: jdk.internal.ref.Cleaner
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:502)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:478)
at java.base/java.lang.Class.forName(Class.java:468)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:84)
... 19 more
Expected behavior
Platform should initialize successfully on JDK 26 and SparkContext creation
should complete. Platform.<clinit> should tolerate the absence of
jdk.internal.ref.Cleaner (e.g. fall back gracefully when the class cannot be
resolved) rather than propagating the ClassNotFoundException as an
ExceptionInInitializerError.
Notes
- Reproduces deterministically on every getOrCreate() — it is not
timing/environment dependent.
- Likely also requires the usual --add-opens/--add-exports JVM flags for
java.base internals on recent JDKs, but the missing class here is a hard
removal, not merely an access restriction, so JVM flags alone do not resolve
it.
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
