Yingjie Cao created FLINK-25781:
-----------------------------------
Summary: Adjust the maximum number of buffers can be used per
result partition for shuffle data read
Key: FLINK-25781
URL: https://issues.apache.org/jira/browse/FLINK-25781
Project: Flink
Issue Type: Sub-task
Components: Runtime / Network
Reporter: Yingjie Cao
Fix For: 1.15.0
In the current sort-shuffle implementation, the maximum number of buffers can
be used per result partition for shuffle data read is 32M. However, for large
parallelism jobs, 32M is not enough and for small parallelism jobs, 32M may
waste buffers. This ticket aims to adjust the maximum number of buffers can be
used per result partition to let it adjust according to parallelism and the
selected value is an empirical one based on the TPC-DS test results.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
