tgravescs commented on issue #26284: [SPARK-29415][Core]Stage Level Sched: Add base ResourceProfile and Request classes URL: https://github.com/apache/spark/pull/26284#issuecomment-553116907 So this doesn't have all the other code, but one thing this first implementation requires is you have to specify the executor resource requirements because requires that for every resourceProfile you get different containers. You could theoretically want 1 set of containers and as long as they have enough resources just go ahead and schedule different tasks on them. Like 1 executor with 4 cpus and 2 gpus. You could scheduler 2 tasks that only require 1 cpu each and then 2 tasks that require 1 cpu and 1 gpu each (@squito example from SPARK-29762). I hadn't given this use case a lot of thought as it introduces more complexity into the scheduling. I think you could potentially still do that with this interface though, it would be spark being smarter about what it allocates. I think it also complicates the tracking quite a bit. Note this current implementation also is not smart about comparing ResourceProfiles. If you create a new ResourceProfile it will assume it needs different containers. We can make it smarter in the future, my first assumption is that many users want to do this so they don't waste resources, so they would simply get 1 container with 2cpu's and then another container with 2cpus and 2 gpus for the tasks in the example above.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
