sunchao commented on PR #57484: URL: https://github.com/apache/spark/pull/57484#issuecomment-5074726475
@pan3793 Interesting! I didn’t realize there was already another PR solving the same problem. It probably makes sense to consolidate the efforts rather than have two similar implementations. WDYT, @starcatmeow? @cloud-fan Yes, the main motivation is to make dynamic allocation more efficient. Bin-packing seemed like a relatively straightforward way to do that while building on Spark’s existing scheduling and scale-down mechanisms. > one idea can be: when the auto scaler predicate an upcoming idle time window, it can pick some executors and mark them as decommissioned, then scheduler will not assign new tasks to these executors, and auto scaler can kill them after current tasks are done. That’s an interesting idea. One question I have is how reliably Spark can predict future executor demand or an upcoming idle window. If demand picks up again shortly afterward, prematurely decommissioning executors could cause unnecessary churn. I also see bin-packing as complementary to that approach: it concentrates tasks on fewer executors, leaving the others idle and creating more opportunities for scale-down. The autoscaler can then decide whether to remove those executors using its existing idle timeout or, in the future, a more demand-aware policy. Happy to have more design discussions and hear more ideas on making auto-scaling better! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
