cloud-fan commented on PR #57484: URL: https://github.com/apache/spark/pull/57484#issuecomment-5071393959
So the motivation seems to make auto-scale more efficient? Can we have a proper design discussion for the end to end auto-scale workflow? Why changing the scheduler is the best design choice? Just for example, one idea can be: when the auto scaler predicate an upcoming idle time window, it can pick some executors and mark them as decommissioned, then scheduler will not assign new tasks to these executors, and auto scaler can kill them after current tasks are done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
