Yingjie Cao created FLINK-22153:
-----------------------------------
Summary: Manually test the sort-merge blocking shuffle
Key: FLINK-22153
URL: https://issues.apache.org/jira/browse/FLINK-22153
Project: Flink
Issue Type: Task
Components: Runtime / Network
Affects Versions: 1.13.0
Reporter: Yingjie Cao
Fix For: 1.13.0
In 1.12, we introduced sort-merge blocking shuffle to Flink and in 1.13, the
feature was optimized which improves the usability (fix direct memory OOM
issue) and performance (introduce IO scheduling and broadcast optimization).
The sort-merge blocking shuffle can be tested following the bellow process:
# Write a simple batch job using either sql/table or DataStream API; (Word
count should be enough)
# Enable sort-merge blocking shuffle by setting
taskmanager.network.sort-shuffle.min-parallelism to 1 in the Flink
configuration file;
# Submit and run the batch job with different parallelism and data volume;
# Tune the relevant config options
(taskmanager.network.blocking-shuffle.compression.enabled,
taskmanager.network.sort-shuffle.min-buffers,
taskmanager.memory.framework.off-heap.batch-shuffle.size) and see the
influence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
