Wan Kun created SPARK-39072:
-------------------------------
Summary: Fast Fail the remaining push blocks if shuffle stage
finalized
Key: SPARK-39072
URL: https://issues.apache.org/jira/browse/SPARK-39072
Project: Spark
Issue Type: Improvement
Components: Shuffle
Affects Versions: 3.3.0
Reporter: Wan Kun
Map task will try to push all map outputs to external shuffle service now.
After the shuffle stage is finalized, the reduce fetch blocks RPC will be
blocked if there are still many map output blocks in flight.
We could stop pushing the remaining blocks if the shuffle stage is finalized.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
