[
https://issues.apache.org/jira/browse/SPARK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392361#comment-14392361
]
Sean Owen commented on SPARK-6662:
----------------------------------
This is kind of a subset of https://issues.apache.org/jira/browse/SPARK-6214 .
I am somewhat concerned about adding a narrow bit of support for one particular
substitution, which in turn is to support a specific assumption in one type of
deployment. Wouldn't you be able to query for the YARN RM address somewhere and
include it in the config?
> Allow variable substitution in spark.yarn.historyServer.address
> ---------------------------------------------------------------
>
> Key: SPARK-6662
> URL: https://issues.apache.org/jira/browse/SPARK-6662
> Project: Spark
> Issue Type: Wish
> Components: YARN
> Affects Versions: 1.3.0
> Reporter: Cheolsoo Park
> Priority: Minor
> Labels: yarn
>
> In Spark on YARN, explicit hostname and port number need to be set for
> "spark.yarn.historyServer.address" in SparkConf to make the HISTORY link. If
> the history server address is known and static, this is usually not a problem.
> But in cloud, that is usually not true. Particularly in EMR, the history
> server always runs on the same node as with RM. So I could simply set it to
> {{$\{yarn.resourcemanager.hostname\}:18080}} if variable substitution is
> allowed.
> In fact, Hadoop configuration already implements variable substitution, so if
> this property is read via YarnConf, this can be easily achievable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
