yuj commented on a change in pull request #29934:
URL: https://github.com/apache/spark/pull/29934#discussion_r499724333
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala
##########
@@ -86,6 +86,7 @@ private[spark] object Constants {
val DEFAULT_EXECUTOR_CONTAINER_NAME = "spark-kubernetes-executor"
val MEMORY_OVERHEAD_MIN_MIB = 384L
val NON_JVM_MEMORY_OVERHEAD_FACTOR = 0.4d
+ val KUBERNETES_MAX_NAME_LENGTH = 63
Review comment:
By the way @stijndehaes I haven't been able to find the document that
says the max name length of any configmap is 63 char long. From my quick
command line test, I found it can be well over beyond that number. Could you
confirm this limit is needed. Well, I do agree it is always good to be
defensive but I found there is no such shortening logic for driver configmap
either. So, could you confirm? Thanks.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/PodTemplateConfigMapStep.scala
##########
@@ -22,23 +22,39 @@ import java.nio.charset.StandardCharsets
import com.google.common.io.Files
import io.fabric8.kubernetes.api.model.{ConfigMapBuilder, ContainerBuilder,
HasMetadata, PodBuilder}
-import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
+import org.apache.spark.deploy.k8s.{KubernetesConf, KubernetesUtils, SparkPod}
import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.{Clock, SystemClock}
-private[spark] class PodTemplateConfigMapStep(conf: KubernetesConf)
- extends KubernetesFeatureConfigStep {
+private[spark] class PodTemplateConfigMapStep
+(conf: KubernetesConf, clock: Clock = new SystemClock())
+ extends KubernetesFeatureConfigStep with Logging {
private val hasTemplate = conf.contains(KUBERNETES_EXECUTOR_PODTEMPLATE_FILE)
+ private val preferredConfigmapName =
s"${conf.resourceNamePrefix}-$POD_TEMPLATE_CONFIGMAP"
+ private val resolvedConfigmapName =
+ if (preferredConfigmapName.length <= KUBERNETES_MAX_NAME_LENGTH) {
+ preferredConfigmapName
Review comment:
By the way @stijndehaes I haven't been able to find the document that
says the max name length of any configmap is 63 char long. From my quick
command line test, I found it can be well over beyond that number. Could you
confirm this limit is needed. Well, I do agree it is always good to be
defensive but I found there is no such shortening logic for driver configmap
either. So, could you confirm? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
