Ngone51 commented on issue #23377: [SPARK-26439][CORE][WIP] Introduce WorkerOffer reservation mechanism for Barrier TaskSet URL: https://github.com/apache/spark/pull/23377#issuecomment-451144043 Hi @squito , I just did a shallow research into YARN gang scheduling. As far as I relazied, YARN do not have a public API directly for gang scheduling, but using its Reservation System(which [have gang semantics](http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html)) to work around gang scheduing (but is not designed for it) at **AM** level. And there's key difference that *[The goal of barrier scheduler is to offer gang semantics in the task scheduling level, whereas the gang semantics in the YARN level is more regarding to resource level.](https://issues.apache.org/jira/browse/SPARK-24723?focusedCommentId=16533205&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16533205)*
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
