holdenk commented on a change in pull request #33508:
URL: https://github.com/apache/spark/pull/33508#discussion_r682842336
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -260,16 +267,22 @@ private[spark] class BasicExecutorFeatureStep(
.withUid(pod.getMetadata.getUid)
.build()
}
+
+ val policy = kubernetesConf.get(KUBERNETES_ALLOCATION_PODSALLOCATOR) match
{
+ case "statefulset" => "Always"
+ case _ => "Never"
+ }
val executorPodBuilder = new PodBuilder(pod.pod)
.editOrNewMetadata()
.withName(name)
.addToLabels(kubernetesConf.labels.asJava)
+ .addToLabels(SPARK_RESOURCE_PROFILE_ID_LABEL,
resourceProfile.id.toString)
Review comment:
Dug through the code looks like this is a duplicate change already
handled in KubeConf so I'll drop it.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -323,6 +323,15 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_ALLOCATION_PODSALLOCATOR =
+ ConfigBuilder("spark.kubernetes.allocation.podsallocator")
+ .doc("Allocator to use for pods. Possible values are direct (the
default) and statefulset." +
+ "Future version may add Job or replicaset.")
+ .version("3.3.0")
+ .stringConf
+ .checkValues(Set("direct", "statefulset"))
Review comment:
Ok works for me :)
##########
File path:
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
##########
@@ -79,6 +79,14 @@ case "$1" in
;;
executor)
shift 1
+ # If the execid is set to the generic EXECID parse the pod name
+ if [ "${SPARK_EXECUTOR_ID}" == "EXECID" ]; then
Review comment:
That didn't seem to be a good match since our executor snapshot depends
on being able to determine the exec id for a given pod as well and if we did it
fully dynamically we wouldn't be able to know and we'd have to make even bigger
changes to how we handle snapshot updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
