GitHub user lianhuiwang opened a pull request:
https://github.com/apache/spark/pull/5580
[SPARK-6869][PySpark] Add pyspark archives path to PYTHONPATH
Based on https://github.com/apache/spark/pull/5478 that provide a
PYSPARK_ARCHIVES_PATH env. from this PR, we just should export
PYSPARK_ARCHIVES_PATH=/user/spark/pyspark.zip,/user/spark/python/lib/py4j-0.8.2.1-src.zip
in conf/spark-env.sh when we don't install PySpark on each node of Yarn. i run
python application successfully on yarn-client and yarn-cluster with this PR.
@andrewor14 @sryza @Sephiroth-Lin Can you take a look at this?thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lianhuiwang/spark SPARK-6869
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5580.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5580
----
commit f72987cc7528e55f8ea43177630a14975b5ecf4f
Author: Lianhui Wang <[email protected]>
Date: 2015-04-16T16:15:49Z
add archives path to PYTHONPATH
commit 9f31dace1fc567db4c12705c9ab7c8fae56a30e5
Author: Lianhui Wang <[email protected]>
Date: 2015-04-19T14:22:45Z
update code and add comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
