[
https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489338#comment-16489338
]
Marcelo Vanzin commented on SPARK-21945:
----------------------------------------
It happens because the import happens before the context is initialized, and
your fix only copies the files during initialization of the context.
To fix this case you'd have to add logic to perform the copy into the launcher
library, which would be kinda weird...
> pyspark --py-files doesn't work in yarn client mode
> ---------------------------------------------------
>
> Key: SPARK-21945
> URL: https://issues.apache.org/jira/browse/SPARK-21945
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.2.0
> Reporter: Thomas Graves
> Assignee: Hyukjin Kwon
> Priority: Major
> Fix For: 2.3.1, 2.4.0
>
>
> I tried running pyspark with --py-files pythonfiles.zip but it doesn't
> properly add the zip file to the PYTHONPATH.
> I can work around by exporting PYTHONPATH.
> Looking in SparkSubmitCommandBuilder.buildPySparkShellCommand I don't see
> this supported at all. If that is the case perhaps it should be moved to
> improvement.
> Note it works via spark-submit in both client and cluster mode to run python
> script.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
