wuyi created SPARK-36530:
----------------------------
Summary: Avoid finalizing when there's no push at all in a shuffle
Key: SPARK-36530
URL: https://issues.apache.org/jira/browse/SPARK-36530
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 3.2.0, 3.3.0
Reporter: wuyi
When all partition data of a map output is bigger thanĀ
spark.shuffle.push.maxBlockSizeToPush, there will be no push. When all map
outputs don't have push, the shuffle doesn't have push. In that case, we don't
need to launch the finalizing request.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]