[
https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lantao Jin updated SPARK-22744:
-------------------------------
Description:
In MapReduce, we can get the submit hostname via checking the value of
configuration {{mapreduce.job.submithostname}}. It can help infra team or other
people to do support work and debug. Bu in Spark, the information is not
included.
In a big company, spark infra team can get the submit host list of all
applications in a clusters. It can help them to control the submition, unify
upgrade and find the bad jobs where it submitted.
Then, I suggest to set the submit hostname in spark internal with name
{{spark.submit.hostname}}, spark user can not set it ( any manually set will be
override).
was:
In MapReduce, we can get the submit hostname via checking the value of
configuration {{mapreduce.job.submithostname}}. It can help infra team or other
people to do support work and debug. Bu in Spark, the information is not
included.
Then, I suggest to introduce a new configuration parameter,
{{spark.submit.hostname}}, which will be set automatically by default, but it
also allows to set this to an user defined hostname if needed (e.g, using an
user node instead of Livy server node).
> Set the application submit hostname by default
> ----------------------------------------------
>
> Key: SPARK-22744
> URL: https://issues.apache.org/jira/browse/SPARK-22744
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.2.1
> Reporter: Lantao Jin
> Priority: Minor
>
> In MapReduce, we can get the submit hostname via checking the value of
> configuration {{mapreduce.job.submithostname}}. It can help infra team or
> other people to do support work and debug. Bu in Spark, the information is
> not included.
> In a big company, spark infra team can get the submit host list of all
> applications in a clusters. It can help them to control the submition, unify
> upgrade and find the bad jobs where it submitted.
> Then, I suggest to set the submit hostname in spark internal with name
> {{spark.submit.hostname}}, spark user can not set it ( any manually set will
> be override).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
