GitHub user buckheroux opened a pull request:
https://github.com/apache/spark/pull/4743
[SPARK-5929] Pyspark: Register a pip requirements file with spark_context
https://issues.apache.org/jira/browse/SPARK-5929
Register a pip requirements file with the spark_context which will ship all
defined dependencies to the workers. Functions similarly to addPyFile.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/buckheroux/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4743.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4743
----
commit 0ed060df2ec5a1a0427df6c160bd51c7014b29da
Author: buck heroux <[email protected]>
Date: 2015-02-18T17:50:27Z
added requirements file to pyspark
commit 6b8bcde60378b58998f5c14d81d72de81f44d718
Author: buck heroux <[email protected]>
Date: 2015-02-18T23:30:28Z
tarfile has no contextmanager in python2.
commit 2773483ea6cc244cb7de02c7dc184391a94d29e6
Author: buck heroux <[email protected]>
Date: 2015-02-19T01:52:06Z
reqs fix
commit 0371ad9b13f96dcc534d897789ccd32f907d5ed9
Author: buck heroux <[email protected]>
Date: 2015-02-19T02:06:48Z
temp tar file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
