liyinan926 commented on a change in pull request #23322: [SPARK-25922][K8]
Spark Driver/Executor spark-app-selector label mism…
URL: https://github.com/apache/spark/pull/23322#discussion_r241901757
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
##########
@@ -60,6 +62,43 @@ private[spark] class KubernetesClusterSchedulerBackend(
removeExecutor(executorId, reason)
}
+ /**
+ * Get an application ID associated with the job.
+ * This returns the string value of [[appId]] if set, otherwise
+ * the locally-generated ID from the superclass.
+ * @return The application ID
+ */
+
+ var appId: Option[String] = None;
+
+ override def applicationId(): String = {
+
+ appId.map(_.toString).getOrElse {
+ logInfo("Initializing Application ID.")
+ bindApplicationId();
+ appId.get
+ }
+ }
+
+ def bindApplicationId(): Unit = {
+ val appIdString = {
+ val wasSparkSubmittedInClusterMode =
conf.get(KUBERNETES_DRIVER_SUBMIT_CHECK)
+
+ // cluster mode: get appId from driver env
+ if (wasSparkSubmittedInClusterMode) {
Review comment:
Instead of checking `wasSparkSubmittedInClusterMode`, I think you can simply
check `spark.app.id` and use the value of it if it's set and use
`super.applicationId`. In client mode, since `KubernetesClientApplication` is
not invoked, how the application ID is generated doesn't really matter, so
using `super.applicationId` is fine for client mode. So the logic is simple: if
`spark.app.id` is set, use the value of it, otherwise use `super.applicationId`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
