GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/9313
[SPARK-10658][PYSPARK][WIP] Provide add jars to py spark api
This does some work to allow dynamic adding of classes to running PySpark
instance, although it still suffers from some of the restrictions mentioned in
SPARK-5185 but provides a utility method for dealing with that. Should we
eventually fix the class loader used by Py4J then this should continue to work
and we can kill the helper method.
Something to note for reviewers: I'm using the test JAR also used by R, I
could make a copy of it but I figured referencing it would be OK but want to
double check.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-10658-provide-addJars-to-pySpark-API
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9313.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9313
----
commit 5eb089203fb859cebb9564d881e001472d31ffec
Author: Holden Karau <[email protected]>
Date: 2015-10-27T21:19:17Z
Add addJar to PySpark API but it isn't very useful at present
commit 3b8b1b086d486c86c0028c19f1875f9e14ea7a53
Author: Holden Karau <[email protected]>
Date: 2015-10-27T22:47:34Z
Add a helper to update the running class loader when adding classes from
python + add a util function to simplify. Follow up should make Kafakutils &
related use the util function
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
