[
https://issues.apache.org/jira/browse/SPARK-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-1480:
-----------------------------------
Description:
The Spark codebase is not always consistent on which class loader it uses when
classlaoders are explicitly passed to things like serializers. This caused
SPARK-1403 and also causes a bug where when the driver has a modified context
class loader it is not translated correctly in local mode to the (local)
executor.
In most cases what we want is the following behavior:
1. If there is a context classloader on the thread, use that.
2. Otherwise use the classloader that loaded Spark.
We should just have a utility function for this and call that function whenever
we need to get a classloader.
Note that SPARK-1403 is a workaround for this exact problem (it sets the
context class loader because downstream code assumes it is set). Once this gets
fixed in a more general way SPARK-1403 can be reverted.
was:
The Spark codebase is not always consistent on which class loader it uses when
classlaoders are explicitly passed to things like serializers. This caused
SPARK-1403 and also causes a bug where when the driver has a modified context
class loader it is not translated correctly.
In most cases what we want is the following behavior:
1. If there is a context classloader on the thread, use that.
2. Otherwise use the classloader that loaded Spark.
We should just have a utility function for this and call that function whenever
we need to get a classloader.
Note that SPARK-1403 is a workaround for this exact problem (it sets the
context class loader because downstream code assumes it is set). Once this gets
fixed in a more general way SPARK-1403 can be reverted.
> Choose classloader consistently inside of Spark codebase
> --------------------------------------------------------
>
> Key: SPARK-1480
> URL: https://issues.apache.org/jira/browse/SPARK-1480
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Priority: Blocker
> Fix For: 1.0.0
>
>
> The Spark codebase is not always consistent on which class loader it uses
> when classlaoders are explicitly passed to things like serializers. This
> caused SPARK-1403 and also causes a bug where when the driver has a modified
> context class loader it is not translated correctly in local mode to the
> (local) executor.
> In most cases what we want is the following behavior:
> 1. If there is a context classloader on the thread, use that.
> 2. Otherwise use the classloader that loaded Spark.
> We should just have a utility function for this and call that function
> whenever we need to get a classloader.
> Note that SPARK-1403 is a workaround for this exact problem (it sets the
> context class loader because downstream code assumes it is set). Once this
> gets fixed in a more general way SPARK-1403 can be reverted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
