Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/539#issuecomment-42512812
I dug through the code a bunch. For those following this PR, here's a quick
summary. Looks like the way we set Spark app name on YARN is actually quite
convoluted because there are two types of app names:
1. The one observed by the RM
2. The one observed by SparkContext
This PR is mainly concerned with (1) and only on the YARN client mode.
Before the changes, there are two ways of affecting (1), in the order of
decreasing priority.
(a) set `spark.app.name` in spark-defaults
(b) set `SPARK_YARN_APP_NAME` in spark-env
After the changes, this becomes
(a) set `SPARK_YARN_APP_NAME` in spark-env
(b) set `spark.app.name` in SparkConf (through `conf.setAppName`)
(c) set `spark.app.name` in spark-defaults
This guarantees that if an environment variable is explicitly specified,
then the RM should use that name. Otherwise, use whatever SparkContext is using
for its app name, in which case (1) == (2). To me, the semantics are pretty
clear.
Notice however that in either before or after, there is now way for
SparkSubmit's --name parameter to affect either (1) or (2). This is a separate
bug with SparkSubmit that I have summarized in
(SPARK-1755)[https://issues.apache.org/jira/browse/SPARK-1755].
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
