tgravescs commented on issue #26284: [SPARK-29415][Core]Stage Level Sched: Add base ResourceProfile and Request classes URL: https://github.com/apache/spark/pull/26284#issuecomment-554543192 The fractional stuff was added in separate jira from the original resource scheduling stuff. like I mention above its essentially allowing multiple tasks to run on a single resource. All our resources require an address with them. For instance in the case of GPU and a machine has say 4 gpus the id could be 0,1,2,4. If you are using fractional and lets say 0.5. This means that you get 2 tasks per GPU. So 2 tasks would be assigned GPU address 0. What they do with that is up to the task code, but the assumption is that the user knows they configured it this way. This is why having fraction over 0.5 doesn't make sense. At that point you can only have 1 task with a single address so you might as well configure it at 1. It doesn't make sense to have 1 task use 3/4 of one gpu and a second task use 1/4 or one gpu and 1/2 of another. To do that you would need more info in the assignments and I don't know how a program would really use that right now. Thus why we restricted it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
