jerrypeng commented on PR #57092: URL: https://github.com/apache/spark/pull/57092#issuecomment-4909972125
@tgravescs thank you for your detail feedback. I will address them inline. > This is a group with multiple pipelined stages (pipelined group), not normal group with a single stage (ie how it runs today) Or is this specifically specified somewhere that we need to do Gang Scheduling? Yes let me clarify that point. Gang admission applies only to a pipelined (multi-stage) group, not to a normal single-stage stage. For a normal group the behavior is the same as today. > I think this opens up some possibilities in the scheduler but I'm not sure it also addresses being able to have multiple shuffle managers or multiple ways to do the shuffle. If you are mixing groups where one is regular and the other is a pipelined group, within the pipelined group you would want to use the StreamingShuffleManager but in the regular group you would use the regular shuffle manager. I think you can make a MultiShuffleManager like is in the streaming shuffle proposal but I think within that it needs to be able to dynamically change per group. It seems like the ShuffleDependency should have a way to indicate what type of shuffle is required. This might be getting more in the future but thinking about it here if we are defining contracts and new ShuffleDependencies. Great question! I did mention this in the original design sketch for this here: https://github.com/apache/spark/pull/56055#issuecomment-4664810708 """ Pluggable shuffle implementation via config. The engine maps the capability to a concrete ShuffleManager. Today we dispatch between sort and streaming managers on a per-job property; the cleaner version selects per-dependency from incrementalHint, with the implementations configured: ``` spark.shuffle.manager = org.apache.spark.shuffle.sort.SortShuffleManager spark.shuffle.manager.incremental = org.apache.spark.shuffle.streaming.StreamingShuffleManager ``` A shuffle with incrementalHint = true is served by the configured incremental manager; everything else by the default — keeping the scheduler construct generic while the shuffle implementation stays pluggable. """ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
