Venkata krishnan Sowrirajan created SPARK-35036:
---------------------------------------------------
Summary: Improve push based shuffle to work with AQE by fetching
partial map indexes for a reduce partition
Key: SPARK-35036
URL: https://issues.apache.org/jira/browse/SPARK-35036
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 3.1.1
Reporter: Venkata krishnan Sowrirajan
Currently when both Push based shuffle and AQE is enabled and when partial set
of map indexes are requested to MapOutputTracker this is delegated the regular
shuffle instead of push based shuffle reading map blocks. This is because
blocks from mapper in push based shuffle are merged out of order due to which
its hard to only get the matching blocks of the reduce partition for the
requested start and end map indexes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
