tgravescs commented on issue #26284: [SPARK-29415][Core]Stage Level Sched: Add base ResourceProfile and Request classes URL: https://github.com/apache/spark/pull/26284#issuecomment-553481259 I think that matching resources to existing Executors complicates a lot of things. Personally, I think that actually makes it harder for the user to reason about it. In your scenario if I have executors with 9GB and then have a stage that needs 8GB, then yes I may want to release the old executors because each one is using an additional 1GB. If I have 1000 executors, that is 1000GB that some other application could be using. Now its possible that I don't want to release those either if it takes a long time to get new ones. But the user can always just create one ResourceProfile with 9GB and use it for both. The only way that doesn't work is if you wanted to change just task requirements for some other resource across them. I think if we clearly document it saying a new ResourceProfile means new executors with the resources specified, then it should be clear. If we don't do that, then I think it is more effort for the user to try to reason about what spark would internally be doing. You would also need an option here that says don't reuse executors and get me new ones for when users don't want to waste resources by reusing executors. I think all of that could be added later, the main first target for this is the ETL/ML use case where in most cases you would want new executors because you want your ETL to run on executors without GPUs and probably have more cores and then when you go to the ML side you would have less cores and a GPU on the executors.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
