dongjoon-hyun opened a new pull request, #59180: URL: https://github.com/apache/spark/pull/59180
### What changes were proposed in this pull request? This PR aims to support `ExecutorResizePlugin` with the `deployment` pods allocator in addition to the default `direct` pods allocator. ### Why are the changes needed? [SPARK-56670](https://github.com/apache/spark/pull/55615) restricted `ExecutorResizePlugin` to the `direct` pods allocator because the higher-level controller's pod template is the source of truth for the other allocators. However, this is not a concern for `Deployment`. Spark never updates its pod template, and the `ReplicaSet` controller neither reverts in-place pod resizes nor rolls out a new revision for them. ### Does this PR introduce _any_ user-facing change? Yes. In Apache Spark 4.2.0, `ExecutorResizePlugin` logs a warning and does not start with the `deployment` pods allocator. After this PR, it works with the `deployment` pods allocator. ### How was this patch tested? Pass the CIs with the updated test case. I also verified manually on K8s v1.37.0 that the executor pods of the `Deployment` are resized in place up to `spark.kubernetes.executor.resizeMaxMemory` without restarts, while the `Deployment` keeps its pod template and a single `ReplicaSet`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
