Repository: spark Updated Branches: refs/heads/master 14d08b990 -> 0451b0014
[SPARK-10986][MESOS] Set the context class loader in the Mesos executor backend. See [SPARK-10986](https://issues.apache.org/jira/browse/SPARK-10986) for details. This fixes the `ClassNotFoundException` for Spark classes in the serializer. I am not sure this is the right way to handle the class loader, but I couldn't find any documentation on how the context class loader is used and who relies on it. It seems at least the serializer uses it to instantiate classes during deserialization. I am open to suggestions (I tried this fix on a real Mesos cluster and it *does* fix the issue). tnachen andrewor14 Author: Iulian Dragos <jagua...@gmail.com> Closes #9282 from dragos/issue/mesos-classloader. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0451b001 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0451b001 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0451b001 Branch: refs/heads/master Commit: 0451b00148a294c665146563242d2fe2de943a02 Parents: 14d08b9 Author: Iulian Dragos <jagua...@gmail.com> Authored: Fri Oct 30 16:51:32 2015 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Fri Oct 30 16:51:32 2015 +0000 ---------------------------------------------------------------------- .../scala/org/apache/spark/executor/MesosExecutorBackend.scala | 5 +++++ 1 file changed, 5 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/0451b001/core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala b/core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala index 0474fd2..c9f18eb 100644 --- a/core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala +++ b/core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala @@ -63,6 +63,11 @@ private[spark] class MesosExecutorBackend logInfo(s"Registered with Mesos as executor ID $executorId with $cpusPerTask cpus") this.driver = driver + // Set a context class loader to be picked up by the serializer. Without this call + // the serializer would default to the null class loader, and fail to find Spark classes + // See SPARK-10986. + Thread.currentThread().setContextClassLoader(this.getClass.getClassLoader) + val properties = Utils.deserialize[Array[(String, String)]](executorInfo.getData.toByteArray) ++ Seq[(String, String)](("spark.app.id", frameworkInfo.getId.getValue)) val conf = new SparkConf(loadDefaults = true).setAll(properties) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
