shivadarshan-devadiga opened a new pull request, #55862:
URL: https://github.com/apache/spark/pull/55862
### What changes were proposed in this pull request?
This PR fixes SPARK-56845 by adding a new configMapName(prefix, suffix)
overload to KubernetesClientUtils that automatically handles ConfigMap names
exceeding Kubernetes' 253-character DNS subdomain limit.
Changes made:
Added new method in KubernetesClientUtils.scala:
- configMapName(prefix: String, suffix: String): Returns prefix + suffix if
≤253 chars, otherwise falls back to spark-<uniqueID><suffix> with a warning log
Updated three feature step classes to use the new method:
- HadoopConfDriverFeatureStep.scala (line 57): Changed from
s"${conf.resourceNamePrefix}-hadoop-config" to
KubernetesClientUtils.configMapName(conf.resourceNamePrefix, "-hadoop-config")
- KerberosConfDriverFeatureStep.scala (line 123): Changed from
s"${kubernetesConf.resourceNamePrefix}-krb5-file" to
KubernetesClientUtils.configMapName(kubernetesConf.resourceNamePrefix,
"-krb5-file")
- PodTemplateConfigMapStep.scala (line 37): Changed from
s"${conf.resourceNamePrefix}-$POD_TEMPLATE_CONFIGMAP" to
KubernetesClientUtils.configMapName(conf.resourceNamePrefix,
s"-$POD_TEMPLATE_CONFIGMAP")
Added comprehensive tests in KubernetesClientUtilsSuite.scala:
- 4 new test cases covering normal operation, boundary conditions (exactly
253 chars), fallback behavior (>253 chars) and backward compatibility
### Why are the changes needed?
When spark.app.name is very long (>222 characters), the derived
resourceNamePrefix combined with fixed suffixes (-hadoop-config, -krb5-file,
-driver-podspec-conf-map) exceeds Kubernetes' DNS subdomain name limit of 253
characters. This causes driver submission to fail with the error: "must be no
more than 253 characters".
The existing KubernetesClientUtils.configMapName(prefix) helper had
truncation logic but was hardcoded to the -conf-map suffix and unused by these
three call sites. This PR generalizes the solution to work with any suffix and
migrates all three problematic call sites to use it.
Why fallback instead of truncation:
- Truncating the prefix could create name collisions when multiple
applications have similar long names
- Using a unique ID (spark-<uniqueID><suffix>) ensures each ConfigMap has a
distinct, valid name
- A warning log alerts users when fallback occurs, maintaining observability
### Does this PR introduce _any_ user-facing change?
Yes, but only for edge cases with very long application names:
Previous behavior:
- Driver submission failed with "must be no more than 253 characters" error
when spark.app.name was very long (>222 chars).
New behavior:
- ConfigMap names that would exceed 253 characters now automatically fall
back to spark-<uniqueID><suffix> format
- A warning is logged: "ConfigMap name '<original-name>' exceeds the maximum
length of 253 characters. Using fallback name '<fallback-name>' instead."
- Driver submission succeeds instead of failing
- For applications with normal-length names (<222 chars), there is no change
in behavior.
### How was this patch tested?
Unit tests added in KubernetesClientUtilsSuite.scala:
- configMapName with custom suffix truncates correctly - Tests short names
pass through, names at 253-char boundary, and fallback for names exceeding limit
- configMapName with different suffixes - Validates all three production
suffixes (-hadoop-config, -krb5-file, -driver-podspec-conf-map)
- configMapName fallback when prefix+suffix exceeds limit - Tests edge cases
with different prefix lengths
- configMapName backward compatibility with default suffix - Ensures
existing configMapName(prefix) calls still work
Test execution:
Ran the full Kubernetes test suite:
Result: 348 tests passed, 0 failed (including the 4 new SPARK-56845 tests)
Test coverage:
- Normal operation (names within limit)
- Boundary condition (exactly 253 characters)
- Fallback behavior (names exceeding limit)
- All three production suffixes
- Backward compatibility with existing code
### Was this patch authored or co-authored using generative AI tooling?
Co-authored -by: Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
