HwangDongJun commented on code in PR #58187: URL: https://github.com/apache/spark/pull/58187#discussion_r4345305984
########## docs/core-migration-guide.md: ########## @@ -26,6 +26,8 @@ license: | - Since Spark 4.4, reading an RDD checkpoint ignores names beginning with `part-` unless the suffix consists of one or more ASCII digits. Previously, names such as `part-00000.bak`, `part-backup`, or `part-` caused checkpoint recovery to fail with `NumberFormatException`. Validation of numeric partition names and the contiguous partition sequence is unchanged. +- Since Spark 4.4, in Kubernetes cluster mode `spark-submit` creates the driver pod with the scheduling gate `spark.apache.org/driver-resources` and removes it once the resources the driver mounts exist, so the driver can no longer start before its config map. Removing the gate requires the `patch` verb on pods, which the account running `spark-submit` may not have; without it the submission fails and the driver pod is deleted. A driver pod whose template sets `spec.nodeName` is not gated and behaves as before. To restore the legacy behavior, you can set `spark.kubernetes.submission.useDriverSchedulingGate` to `false`, or grant `patch` on pods to the account that runs `spark-submit`. Review Comment: Since the default is `false` now (see the `Config.scala` thread), this change no longer alters the behavior of an upgraded deployment, so I removed the entry from this PR instead of rewording it. The `patch` requirement, the scheduler requirement and the `nodeName` case are described in `docs/running-on-kubernetes.md` as conditions of enabling the setting. Your suggested text can go into the migration guide entry of the change that flips the default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
