GitHub user piaozhexiu opened a pull request:
https://github.com/apache/spark/pull/5321
[SPARK-6662][YARN] Allow variable substitution in
spark.yarn.historyServer.address
In Spark on YARN, explicit hostname and port number need to be set for
"spark.yarn.historyServer.address" in SparkConf to make the HISTORY link. If
the history server address is known and static, this is usually not a problem.
But in cloud, that is usually not true. Particularly in EMR, the history
server always runs on the same node as with RM. So I could simply set it to
${yarn.resourcemanager.hostname}:18080 if variable substitution is allowed.
In fact, Hadoop configuration already implements variable substitution, so
if this property is read via YarnConf, this can be easily achievable.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/piaozhexiu/spark SPARK-6662
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5321.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5321
----
commit ff9c35d2fdf08045f51b3686158bb10df9383651
Author: Cheolsoo Park <[email protected]>
Date: 2015-04-01T23:33:38Z
Allow variable substitution for spark.yarn. properties
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
