GitHub user buckheroux opened a pull request:
https://github.com/apache/spark/pull/4897
[SPARK-5929] Pyspark: Register a pip requirements file with spark_context
Ships all packages in the requirements file by installing them locally via
pip and then ships the packages to the workers via addPyFile
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/buckheroux/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4897.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4897
----
commit 0ed060df2ec5a1a0427df6c160bd51c7014b29da
Author: buck heroux <[email protected]>
Date: 2015-02-18T17:50:27Z
added requirements file to pyspark
commit 6b8bcde60378b58998f5c14d81d72de81f44d718
Author: buck heroux <[email protected]>
Date: 2015-02-18T23:30:28Z
tarfile has no contextmanager in python2.
commit 2773483ea6cc244cb7de02c7dc184391a94d29e6
Author: buck heroux <[email protected]>
Date: 2015-02-19T01:52:06Z
reqs fix
commit 0371ad9b13f96dcc534d897789ccd32f907d5ed9
Author: buck heroux <[email protected]>
Date: 2015-02-19T02:06:48Z
temp tar file
commit f2a46e5d6e309a5ba29259cc1f77e594d932b0f5
Author: buck heroux <[email protected]>
Date: 2015-03-05T00:51:46Z
bubbled up try finally
commit fca4be61c6542b807a0d5370f761ef031fc7eb86
Author: buck heroux <[email protected]>
Date: 2015-03-05T00:53:17Z
forgot to remove
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
