[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90704/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90704/testReport)** for PR 21267 at commit [`ef3555e`](https://github.com/apache/spark/commit/ef3555e389ea36159e9a1dfd076e9f6afbaf3f35). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3277/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90704/testReport)** for PR 21267 at commit [`ef3555e`](https://github.com/apache/spark/commit/ef3555e389ea36159e9a1dfd076e9f6afbaf3f35). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90616/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90616/testReport)** for PR 21267 at commit [`ef3555e`](https://github.com/apache/spark/commit/ef3555e389ea36159e9a1dfd076e9f6afbaf3f35). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90614/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90614/testReport)** for PR 21267 at commit [`b9e312e`](https://github.com/apache/spark/commit/b9e312ecfd0215c669e1826e891ccbaa5937ea49). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3214/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90616/testReport)** for PR 21267 at commit [`ef3555e`](https://github.com/apache/spark/commit/ef3555e389ea36159e9a1dfd076e9f6afbaf3f35). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3213/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90614/testReport)** for PR 21267 at commit [`b9e312e`](https://github.com/apache/spark/commit/b9e312ecfd0215c669e1826e891ccbaa5937ea49). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21267 Looks good aside from the log message. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90565/testReport)** for PR 21267 at commit [`68be3ba`](https://github.com/apache/spark/commit/68be3baef22d8b7aa58a432cb5bd12437c07feb7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90565/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3184/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90565/testReport)** for PR 21267 at commit [`68be3ba`](https://github.com/apache/spark/commit/68be3baef22d8b7aa58a432cb5bd12437c07feb7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 Hm .. @jerryshao, seems it's a bit difficult to do so. The simplest way should be just to copy files into the directories in `SparkFiles.getRootDirectory`; however, `SparkEnv` is inaccessible at this stage in `SparkSubmit` .. Another way might be to find if there's a way by setting `spark.files` so that they are added via `addFile` later which put the file in `SparkFiles.getRootDirectory` at driver side too but .. I wonder if it makes sense to set this which Yarn doesn't use. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 Will try to put this into SparkSubmit. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 (I have tried to explain why it's specific to PySpark shell with Yarn client mode in PR description) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21267 Yea, this is specific to yarn client PySpark shell. In case of yarn client and cluster with submit, they are specially handled via #6360 but I think PySpark shell in yarn client mode was missed out. The way of launching it is diverted if I understood correctly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21267 Does it only happen in yarn client PySpark shell? I would suggest to fix this in the SparkSubmit side, to treat this as a special case and set the proper config. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21267 **[Test build #90364 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90364/testReport)** for PR 21267 at commit [`68be3ba`](https://github.com/apache/spark/commit/68be3baef22d8b7aa58a432cb5bd12437c07feb7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90364/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21267: [SPARK-21945][YARN][PYTHON] Make --py-files work with Py...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21267 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3036/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
