[
https://issues.apache.org/jira/browse/SPARK-36627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407531#comment-17407531
]
Apache Spark commented on SPARK-36627:
--------------------------------------
User 'fsamuel-bs' has created a pull request for this issue:
https://github.com/apache/spark/pull/33879
> Tasks with Java proxy objects fail to desrialize
> ------------------------------------------------
>
> Key: SPARK-36627
> URL: https://issues.apache.org/jira/browse/SPARK-36627
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.3
> Reporter: Samuel Souza
> Priority: Minor
>
> In JavaSerializer.JavaDeserializationStream we override resolveClass of
> ObjectInputStream to use the threads' contextClassLoader. However, we do not
> override resolveProxyClass, which is used when deserializing Java proxy
> objects, which makes spark use the wrong classloader when deserializing
> objects, which causes the job to fail with the following exception:
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 1.0 (TID 4, <host>, executor 1): java.lang.ClassNotFoundException:
> <class>
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:398)
> at
> java.base/java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:829)
> at
> java.base/java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1917)
> ...
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
