Yuxin Tan created FLINK-32709:
---------------------------------
Summary: Modify segment size to improve memory utilization for
Hybrid Shuffle
Key: FLINK-32709
URL: https://issues.apache.org/jira/browse/FLINK-32709
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Currently, each subpartition in Disk/Remote has a segment size of 8M. When
writing segments to the Disk tier with a parallelism of 1000, only shuffle data
exceeding 1000 * 8M can be written to the Memory tier again. However, for most
shuffles, the data volume size falls below this limit, significantly impacting
Memory tier utilization.
For better performance, it is necessary to address this issue to improve the
memory tier utilization.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
