TongWei1105 commented on PR #59756: URL: https://github.com/apache/spark/pull/59756#issuecomment-6546052665
Thanks for explaining your concerns, @dongjoon-hyun. I'd like to clarify our production scenario and the motivation for this proposal. Our deployment is based on Spark 3.5 and uses dynamic allocation. In our environment, the Kubernetes control plane is not saturated. The limitation we observe is the direct allocator waiting for each Pod creation request to complete before submitting the next one. The intent is to use available control-plane capacity through bounded, opt-in concurrency, not to work around an overloaded control plane. StatefulSet is not a behaviorally equivalent replacement for our workload. In our Spark 3.5 implementation, Spark adjusts the StatefulSet replica count while separately removing the executors selected by dynamic allocation. StatefulSet manages replicas by Pod ordinal and does not necessarily remove the same executors Spark selected. Preserving the existing executor selection behavior during scale-down is important to us. Our current codebase does not include the Deployment allocator. We understand that this version constraint alone does not justify an upstream change, but adopting a different allocator would require migration and validation of its lifecycle behavior. Another motivation is to provide operators with a supported tuning option for corner cases like ours. A custom allocator is a possible workaround, but asking each user encountering this limitation to maintain a separate implementation creates ongoing maintenance and compatibility costs. A shared, opt-in capability could address these cases while retaining the direct allocator and leaving the default behavior unchanged. We acknowledge that bounded concurrency can still increase the API request rate and that the additional implementation complexity needs to be justified. We are not suggesting enabling it for every deployment. Would you be open to considering it as an optional tuning capability for environments with available control-plane capacity where retaining direct allocation semantics is important? If you need further evidence to assess that trade-off, please let us know which measurements would be most useful. Thanks also for the separate implementation review. This reply addresses the motivation; it does not resolve the code-level findings. Thanks for your time and feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
