[
https://issues.apache.org/jira/browse/FLINK-22153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322641#comment-17322641
]
Yingjie Cao commented on FLINK-22153:
-------------------------------------
[~trohrmann] Thanks for testing the sort-merge blocking shuffle and you are
right, I already tested it with tpcds queries on a cluster.
# I will add some logs when sort-merge result partition is initialized.
# I agree that we should increase the default
{{taskmanager.network.sort-shuffle.min-buffers, I will do some test and open a
PR for that.}}
# We already have ticket (FLINK-21967) for documenting blocking shuffle which
should include both the default bounded blocking one and the sort-merge one.
Thanks again for trying out the feature and the good suggestions.
> Manually test the sort-merge blocking shuffle
> ---------------------------------------------
>
> Key: FLINK-22153
> URL: https://issues.apache.org/jira/browse/FLINK-22153
> Project: Flink
> Issue Type: Task
> Components: Runtime / Network
> Affects Versions: 1.13.0
> Reporter: Yingjie Cao
> Assignee: Till Rohrmann
> Priority: Blocker
> Fix For: 1.13.0
>
>
> In 1.12, we introduced sort-merge blocking shuffle to Flink and in 1.13, the
> feature was optimized which improves the usability (fix direct memory OOM
> issue) and performance (introduce IO scheduling and broadcast optimization).
> The sort-merge blocking shuffle can be tested following the bellow process:
> # Write a simple batch job using either sql/table or DataStream API; (Word
> count should be enough)
> # Enable sort-merge blocking shuffle by setting
> taskmanager.network.sort-shuffle.min-parallelism to 1 in the Flink
> configuration file;
> # Submit and run the batch job with different parallelism and data volume;
> # Tune the relevant config options
> (taskmanager.network.blocking-shuffle.compression.enabled,
> taskmanager.network.sort-shuffle.min-buffers,
> taskmanager.memory.framework.off-heap.batch-shuffle.size) and see the
> influence.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
