littlexyw commented on code in PR #53338:
URL: https://github.com/apache/spark/pull/53338#discussion_r2613982382
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala:
##########
@@ -116,6 +116,31 @@ object KubernetesClientUtils extends Logging {
}.toList.sortBy(x => x.getKey) // List is sorted to make mocking based
tests work
}
+ @Since("4.2.0")
+ def overrideDefaultSparkEnv(conf: KubernetesDriverConf,
+ confFilesMap: Map[String, String]): Map[String, String] = {
+ if (conf.environment.isEmpty) {
+ return confFilesMap
+ }
+ val customEnvs = conf.environment.map {
+ case (k, v) => s"export $k=$v"
+ }.mkString("\n", "\n", "\n")
+ val mapSize = confFilesMap.map {
+ case (k, v) => k.length + v.length
+ }.sum
+ val maxSize = conf.sparkConf.get(Config.CONFIG_MAP_MAXSIZE)
+ if (mapSize + customEnvs.length < maxSize) {
+ confFilesMap ++ Map(SPARK_ENV_FILE_NAME ->
+ s"${confFilesMap.getOrElse(SPARK_ENV_FILE_NAME, "")}$customEnvs")
+ } else {
+ logWarning(log"Skipped custom driver env, due to size constraint" +
Review Comment:
> This seems to introduce unsafe behavior to the Spark. Previously, we
didn't fail?
Thanks for catching this! You're absolutely right - the original behavior
didn't fail when it should have. I agree this introduces unsafe behavior.
To address this, I've submitted a separate PR that adds proper validation:
#53455: Fail app on ConfigMap size over limit
Could you please review this new fix when you have a moment? **Would love to
hear your thoughts!**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
