I am trying to convert an application to Spark and need to find out all
the serialization issues. Unfortunately, the SerializationDebugger
appears to no longer work with Java 21 and presumably also Java 17. The
problem is reflective access to sun.security.action.GetBooleanAction,
which is prohibited since Java 17 (https://openjdk.org/jeps/403).
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:50)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:122)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.util.SparkClosureCleaner$.clean(SparkClosureCleaner.scala:42)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.SparkContext.clean(SparkContext.scala:2839)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.rdd.RDD.$anonfun$flatMap$1(RDD.scala:434)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.rdd.RDD.withScope(RDD.scala:417)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.rdd.RDD.flatMap(RDD.scala:433)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.api.java.JavaRDDLike.flatMap(JavaRDDLike.scala:127)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.api.java.JavaRDDLike.flatMap$(JavaRDDLike.scala:125)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.api.java.AbstractJavaRDDLike.flatMap(JavaRDDLike.scala:46)
~[spark-core_2.13-4.0.0.jar:4.0.0]
[...]
Caused by: java.lang.IllegalAccessException: access to public member
failed:
sun.security.action.GetBooleanAction.<init>[Ljava.lang.Object;@3d69a79d/invokeSpecial,
from class org.apache.spark.serializer.SerializationDebugger$ (unnamed
module @343f4d3d)
at
java.base/java.lang.invoke.MemberName.makeAccessException(MemberName.java:894)
~[?:?]
at
java.base/java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:3987)
~[?:?]
at
java.base/java.lang.invoke.MethodHandles$Lookup.getDirectConstructorCommon(MethodHandles.java:4227)
~[?:?]
at
java.base/java.lang.invoke.MethodHandles$Lookup.getDirectConstructorNoSecurityManager(MethodHandles.java:4221)
~[?:?]
at
java.base/java.lang.invoke.MethodHandles$Lookup.unreflectConstructor(MethodHandles.java:3534)
~[?:?]
at
org.apache.spark.serializer.SerializationDebugger$.<clinit>(SerializationDebugger.scala:76)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:50)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:122)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.util.SparkClosureCleaner$.clean(SparkClosureCleaner.scala:42)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.SparkContext.clean(SparkContext.scala:2839)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.rdd.RDD.$anonfun$flatMap$1(RDD.scala:434)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.rdd.RDD.withScope(RDD.scala:417)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at org.apache.spark.rdd.RDD.flatMap(RDD.scala:433)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.api.java.JavaRDDLike.flatMap(JavaRDDLike.scala:127)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.api.java.JavaRDDLike.flatMap$(JavaRDDLike.scala:125)
~[spark-core_2.13-4.0.0.jar:4.0.0]
at
org.apache.spark.api.java.AbstractJavaRDDLike.flatMap(JavaRDDLike.scala:46)
~[spark-core_2.13-4.0.0.jar:4.0.0]
[...]
This is for spark-core_2.13 version 4.0.0.
Unfortunately, I cannot packport my application to an earlier version of
Java. Any help is appreciated.
Clemens
--
Dr. Clemens Ballarin
ANKORDATA GmbH & Co. KG
Schillerstraße 4
60313 Frankfurt
Tel.: +49 69 928879-21
Internet: http://www.ankordata.de
ANKORDATA GmbH & Co. KG, Amtsgericht Frankfurt, HRA 43069
Persönlich haftende Gesellschafterin:
ANKORDATA Verwaltung GmbH, Frankfurt, Amtsgericht Frankfurt, HRB 76831
Geschäftsführer: Michael Ebert, Jens Grüntjes
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
