[
https://issues.apache.org/jira/browse/SPARK-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-1480:
-----------------------------------
Description:
The Spark codebase is not always consistent on which class loader it uses when
classlaoders are explicitly passed to things like serializers. This caused
SPARK-1403 and also causes a bug where when the driver has a modified context
class loader it is not translated correctly.
In most cases what we want is the following behavior:
1. If there is a context classloader on the thread, use that.
2. Otherwise use the classloader that loaded Spark.
We should just have a utility function for this and call that function whenever
we need to get a classloader.
Note that SPARK-1403 is a workaround for this exact problem (it sets the
context class loader because downstream code assumes it is set). Once this gets
fixed in a more general way SPARK-1403 can be reverted.
was:
The Spark codebase is not always consistent on which class loader it uses when
classlaoders are explicitly passed to things like serializers.
In most cases what we want is the following behavior:
1. If there is a context classloader on the thread, use that.
2. Otherwise use the classloader that loaded Spark.
We should just have a utility function for this and call that function whenever
we need to get a classloader.
Note that SPARK-1403 is a workaround for this exact problem (it sets the
context class loader because downstream code assumes it is set). Once this gets
fixed in a more general way SPARK-1403 can be reverted.
> Choose classloader consistently inside of Spark codebase
> --------------------------------------------------------
>
> Key: SPARK-1480
> URL: https://issues.apache.org/jira/browse/SPARK-1480
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Priority: Blocker
> Fix For: 1.0.0
>
>
> The Spark codebase is not always consistent on which class loader it uses
> when classlaoders are explicitly passed to things like serializers. This
> caused SPARK-1403 and also causes a bug where when the driver has a modified
> context class loader it is not translated correctly.
> In most cases what we want is the following behavior:
> 1. If there is a context classloader on the thread, use that.
> 2. Otherwise use the classloader that loaded Spark.
> We should just have a utility function for this and call that function
> whenever we need to get a classloader.
> Note that SPARK-1403 is a workaround for this exact problem (it sets the
> context class loader because downstream code assumes it is set). Once this
> gets fixed in a more general way SPARK-1403 can be reverted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
