[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424825#comment-17424825 ]
Gengliang Wang commented on SPARK-36892: ---------------------------------------- [~mridulm80] [~mshen] [~zhouyejoe] [~apatnam] Again, thanks for testing Spark 3.2 with real workloads. Now that all the blockers are resolved. I will have the new RC soon. > Disable batch fetch for a shuffle when push based shuffle is enabled > -------------------------------------------------------------------- > > Key: SPARK-36892 > URL: https://issues.apache.org/jira/browse/SPARK-36892 > Project: Spark > Issue Type: Bug > Components: Shuffle > Affects Versions: 3.2.0 > Reporter: Mridul Muralidharan > Assignee: Ye Zhou > Priority: Blocker > Fix For: 3.2.0 > > > When push based shuffle is enabled, efficient fetch of merged mapper shuffle > output happens. > Unfortunately, this currently interacts badly with > spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle > fetch to hang and/or duplicate data to be fetched, causing correctness issues. > Given batch fetch does not benefit spark stages reading merged blocks when > push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can > be disabled when push based shuffle is enabled. > Thx to [~Ngone51] for surfacing this issue. > +CC [~Gengliang.Wang] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org