[
https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated SPARK-35671:
----------------------------------
Description:
With push-based shuffle enabled, the reducers send 2 different requests to the
ESS:
1. Request to fetch the metadata of the merged shuffle block.
2. Requests to fetch the data of the merged shuffle block in chunks which are
by default 2MB in size.
The ESS should be able to serve these requests and this Jira targets all the
changes in the network-common and network-shuffle modules to be able to support
this.
was:
With push-based shuffle enabled, the reducers send 2 different requests to the
ESS:
1. Request to fetch the metadata of the merged shuffle block.
2. Requests to fetch the data of the merged shuffle block in chunks which are
by default 2MB in size.
The ESS should be able to serve these requests and this Jira targets all the
changes in the ESS to be able to support this.
> Add Support in the ESS to serve merged shuffle block meta and data to
> executors
> -------------------------------------------------------------------------------
>
> Key: SPARK-35671
> URL: https://issues.apache.org/jira/browse/SPARK-35671
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle
> Affects Versions: 3.1.0
> Reporter: Chandni Singh
> Priority: Major
>
> With push-based shuffle enabled, the reducers send 2 different requests to
> the ESS:
> 1. Request to fetch the metadata of the merged shuffle block.
> 2. Requests to fetch the data of the merged shuffle block in chunks which
> are by default 2MB in size.
> The ESS should be able to serve these requests and this Jira targets all the
> changes in the network-common and network-shuffle modules to be able to
> support this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
