[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mridul Muralidharan updated SPARK-32920: ---------------------------------------- Fix Version/s: 3.2.0 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > ------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core > Affects Versions: 3.1.0 > Reporter: Min Shen > Assignee: Venkata krishnan Sowrirajan > Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org