hehuiyuan commented on a change in pull request #24009: [k8s]Unify the three
variables' name : pod name prefix in kubernetes / spark.app.name in spark
ui / spark-app-name in pod's annotations
URL: https://github.com/apache/spark/pull/24009#discussion_r263628017
##########
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##########
@@ -115,9 +115,13 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
with Logging with Seria
set("spark.master", master)
}
- /** Set a name for your application. Shown in the Spark web UI. */
+ /** Set a name for your application. Shown in the Spark web UI.
+ * For spark on kubernetes,Unify the three variables' name :
Review comment:
/** Set a name for your application. Shown in the Spark web UI.
* For spark on kubernetes,Unify the three variables' name :
* pod name prefix in kubernetes / spark.app.name in spark ui /
spark-app-name in pod's annotations
* */
It uses the value of spark.app.name to name the pod name for spark on k8s
when submitting tasks by spark-submit . Meanwhile KubernetesClientApplication
calls start function ：
`start(args: Array[String], conf: SparkConf)`
in conf parameter, the value of spark.app.name equals the value of "--name".
But the value of spark.app.name is not coincident.
`val conf = new SparkConf().**setAppName("JavaNetworkWordCount")`
` _ui =
if (conf.getBoolean("spark.ui.enabled", true)) {
Some(SparkUI.create(Some(this), _statusStore, _conf,
_env.securityManager, appName, "",
startTime))
} else {
// For tests, do not enable the UI
None
}`
Create an UI by calling SparkUI's create function which contains _conf
parameter.
The value of spark.app.name is setting by calling setAppName method.
KubernetesClientApplication.start and SparkUI.create ,both functions have
SparkConf parameter. They have the possibility of inconsistency for the value
of spark.app.name in SparkConf.
Refer to the pictures and content in the previous comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
