GitHub user taroplus opened a pull request:
https://github.com/apache/spark/pull/17074
[SPARK-18646][REPL] Set parent classloader as null for ExecutorClassLoader
## What changes were proposed in this pull request?
By default, a class loader will have java's system class loader as its
parent, however this ExecutorClassLoader should always resolve classes from
given 'paren't class loader.
Problem happens when an application has a class loader structure like below
System-Classloader [ClassA, Runtime]
|
Application-Classloader [ClassA, Spark, Jackson etc etc] : **paren'**
|
Executor Class Loader
The application serializes ClassA using Application class loader however
Spark deserializes the class using System Class loader, this leads to various
problems.
The change here to pass null to the parent class so that a link between
ExecutorClassLoader and SystemClassLoader no longer exists.
For most of situations, the parent class loader is the system class loader
so this wouldn't have any impact.
## How was this patch tested?
UTs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/taroplus/spark executor_classloader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17074.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17074
----
commit e0d95381ffc70c22bc2c8a27f908d97f565b2566
Author: Kohki Nishio <[email protected]>
Date: 2017-02-26T16:22:03Z
Set parent classloader as null for ExecutorClassLoader
This change is to address a classloader problem under sbt environment
more details can be found in Jira
[https://issues.apache.org/jira/browse/SPARK-19675]
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
