[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489397#comment-16489397 ] Hyukjin Kwon commented on SPARK-21945: -- For another clarification, the launch execution codepath is

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489369#comment-16489369 ] Hyukjin Kwon commented on SPARK-21945: -- Just for clarification, zip file works fine because the

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489367#comment-16489367 ] Hyukjin Kwon commented on SPARK-21945: -- To be more correct, the paths are added as are given my

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489338#comment-16489338 ] Marcelo Vanzin commented on SPARK-21945: It happens because the import happens before the context

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488686#comment-16488686 ] Hyukjin Kwon commented on SPARK-21945: -- Same thing happens in standalone mode too. > pyspark

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488570#comment-16488570 ] Hyukjin Kwon commented on SPARK-21945: -- cc [~jerryshao] FYI. > pyspark --py-files doesn't work in

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488486#comment-16488486 ] Hyukjin Kwon commented on SPARK-21945: -- Right. This case looks weird again. In case the given Python

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487707#comment-16487707 ] Marcelo Vanzin commented on SPARK-21945: {noformat} $ cat test.py from pyspark import

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486610#comment-16486610 ] Hyukjin Kwon commented on SPARK-21945: -- [~vanzin], I tried spark-submit in client mode and cluster

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480199#comment-16480199 ] Dongjoon Hyun commented on SPARK-21945: --- Thank you for verifying and including this in 2.3.1 RC2,

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479765#comment-16479765 ] Marcelo Vanzin commented on SPARK-21945: This works for cluster mode and pyspark (shell), but

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479610#comment-16479610 ] Marcelo Vanzin commented on SPARK-21945: Let me run some tests and if they're ok, I'll cherry

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479574#comment-16479574 ] Dongjoon Hyun commented on SPARK-21945: --- Hi, [~vanzin], [~tgraves], [~hyukjin.kwon]. I'm wondering

[jira] [Commented] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467052#comment-16467052 ] Apache Spark commented on SPARK-21945: -- User 'HyukjinKwon' has created a pull request for this