jerrypeng commented on PR #57092: URL: https://github.com/apache/spark/pull/57092#issuecomment-4909070098
**Reposting @tgravescs feedback from: https://github.com/apache/spark/pull/56055#issuecomment-4908525214** Overall I really like the new idea of having a Pipelined Shuffle Dependency and support it directly in DagScheduler and define the contract. This opens it up to be generally useful in other scenarios. I'm still going through some of the details but here are a few initial comments I think we need to clarify some of the gang scheduling wording. A group is admitted only if the cluster can currently run all tasks of all member stages concurrently; This is a group with multiple pipelined stages (pipelined group), not normal group with a single stage (ie how it runs today) Or is this specifically specified somewhere that we need to do Gang Scheduling? I think this opens up some possibilities in the scheduler but I'm not sure it also addresses being able to have multiple shuffle managers or multiple ways to do the shuffle. If you are mixing groups where one is regular and the other is a pipelined group, within the pipelined group you would want to use the StreamingShuffleManager but in the regular group you would use the regular shuffle manager. I think you can make a MultiShuffleManager like is in the streaming shuffle proposal but I think within that it needs to be able to dynamically change per group. It seems like the ShuffleDependency should have a way to indicate what type of shuffle is required. This might be getting more in the future but thinking about it here if we are defining contracts and new ShuffleDependencies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
