GitHub user GrahamDennis opened a pull request:
https://github.com/apache/spark/pull/1890
[SPARK-2878]: Fix custom spark.kryo.registrator
This is a work-in-progress, and I'm looking for feedback on my current
approach. My aim here is to add the user jars specified in SparkConf into the
Executor processes before they start any tasks, and to add these jars to the
class path of all threads.
So far I have just implemented sending the user jars to the Executor
processes before registering the executor with the master. Next steps are to
add these jars to all threads, and/or share a pool of kryo instances created
with a known class loader. This will also need implementing for Mesos & YARN.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/GrahamDennis/spark feature/spark-2878
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1890.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1890
----
commit 8cb52099c0faa04234cb2741d98d2fb689f6a42c
Author: Graham Dennis <[email protected]>
Date: 2014-08-11T11:05:07Z
[WIP][SPARK-2878]: Fix custom spark.kryo.registrator
My aim here is to add the user jars specified in SparkConf into the
Executor processes before they launch any tasks, and add these jars to the
class path of all threads.
So far I have just implemented sending the user jars to the Executor
processes before registering the executor with the master. Next steps are to
add these jars to all threads, and/or share a pool of kryo instances created
with a known class loader. This will also need implementing for Mesos & YARN.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
