Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/5580#issuecomment-96369904
@andrewor14 for second question,i add two things for it.one is i add zip
pyspark archives to pyspark/lib when we build spark jar. other is in submit if
PYSPARK_ARCHIVES_PATH does not exist and pyspark.zip does not exist, then we
zip archives to pyspark/lib.
and i add a conf 'spark.submit.pyArchives' to store pyspark archives that
let Client know pyArchives.we cannot use PYSPARK_ARCHIVES_PATH env because
Client and spark-submit is one process and when we set PYSPARK_ARCHIVES_PATH in
submit, Client cannot get it.
One thing to note that if we install PYSPARKPATH on every node, now we need
to set local pyspark archives to PYSPARK_ARCHIVES_PATH. because spark-submit
will check whether PYSPARK_ARCHIVES_PATH exists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
