Ngone51 commented on issue #23377: [SPARK-26439][CORE][WIP] Introduce WorkerOffer reservation mechanism for Barrier TaskSet URL: https://github.com/apache/spark/pull/23377#issuecomment-451041130 > such a big change to the scheduler It may be a big change for Barrier TaskSet, but remains no difference for non-barrier TaskSet yet. > by itself the improvement here is just for having multiple concurrent barrier stages, still with static allocation? Yes, and I think the improvement here could also handle a mix sitution of non-barrier & barrier TaskSets. Dynamic allocation would be the next step if this change could be accepted. > Have you looked at gang scheduling in yarn? Sorry, no. But I know YARN has reservation mechanism for resource, too. I'll have a look for YARN gang scheduling. > I think at this stage, we might benefit more from a high-level design discussion than from code? Agreed. And it would be better if we could get more people in. Let me try @jiangxb1987 @jerryshao
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
