wbo4958 opened a new pull request, #50475:
URL: https://github.com/apache/spark/pull/50475
### What changes were proposed in this pull request?
This PR is to construct the session-specific classloader based on the
default session classloader which has already added the global jars (e.g.,
added by `--jars` ) into the classpath on the executor side in the connect mode.
### Why are the changes needed?
In Spark Connect mode, when connecting to a non-local (e.g., standalone)
cluster, the executor creates an isolated session state that includes a
session-specific classloader for each task. However, a notable issue arises:
this session-specific classloader does not include the global JARs specified by
the --jars option in the classpath. This oversight can lead to deserialization
exceptions. For example:
``` console
Caused by: java.lang.ClassCastException: cannot assign instance of
java.lang.invoke.SerializedLambda to field
org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of
org.apache.spark.rdd.MapPartitionsRDD
at
java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2096)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
The newly added test can pass.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
