Chandni Singh created SPARK-35671:
-------------------------------------
Summary: Add Support in the ESS to serve merged shuffle block meta
and data to executors
Key: SPARK-35671
URL: https://issues.apache.org/jira/browse/SPARK-35671
Project: Spark
Issue Type: Sub-task
Components: Shuffle
Affects Versions: 3.1.0
Reporter: Chandni Singh
With push-based shuffle enabled, the reducers send 2 different requests to the
ESS:
1. Request to fetch the metadata of the merged shuffle block.
2. Requests to fetch the data of the merged shuffle block in chunks which are
by default 2MB in size.
The ESS should be able to serve these requests and this Jira targets all the
changes in the ESS to be able to support this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
