GitHub user jerryshao opened a pull request:

    [SPARK-17512][Core] Avoid formatting to python path for yarn and mesos 
cluster mode

    ## What changes were proposed in this pull request?
    Yarn and mesos cluster mode support remote python path (HDFS/S3 scheme) by 
their own mechanism, it is not necessary to check and format the python when 
running on these modes. This is a potential regression compared to 1.6, so here 
propose to fix it.
    ## How was this patch tested?
    Unit test to verify SparkSubmit arguments, also with local cluster 
verification. Because of lack of `MiniDFSCluster` support in Spark unit test, 
there's no integration test added.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-17512

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15137
commit 2051d46a1d983a591086722517faad507e467bb7
Author: jerryshao <>
Date:   2016-09-18T07:14:41Z

    Avoid formatting path for yarn and mesos cluster mode

commit d136155a734d1e779faa96b0a864c3ded739141f
Author: jerryshao <>
Date:   2016-09-18T07:30:49Z

    add unittest


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to