GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/699
[SPARK-1755] Respect SparkSubmit --name on YARN
Right now, SparkSubmit ignores the `--name` flag for both yarn-client and
yarn-cluster. This is a bug.
- In client mode, SparkSubmit treats `--name` as a cluster config and does
not propagate this to SparkContext.
- In cluster mode, SparkSubmit passes this flag to
`org.apache.spark.deploy.yarn.Client`, which only uses it for the YARN
ResourceManager, but does not propagate this to SparkContext.
This PR ensures that `spark.app.name` is always set if SparkSubmit receives
the `--name` flag, which is what the usage promises. This makes it possible for
applications to start a SparkContext with an empty conf `val sc = new
SparkContext(new SparkConf)`, and inherit the app name from SparkSubmit.
Tested both modes on a YARN cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark yarn-app-name
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/699.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #699
----
commit c86d9ca7747d8eede9d84342413861b17ade27af
Author: Andrew Or <[email protected]>
Date: 2014-05-08T18:45:35Z
Respect SparkSubmit --name on YARN
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
