[
https://issues.apache.org/jira/browse/FLINK-31079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691969#comment-17691969
]
Lijie Wang commented on FLINK-31079:
------------------------------------
Thanks [~lsy]. Currently, the
[{{execution.batch.adaptive.auto-parallelism.avg-data-volume-per-task}}|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#execution-batch-adaptive-auto-parallelism-avg-data-volume-per-ta]
is cluster level, but I personally think it makes sense to make it a job
level, especially when using session cluster, I will evaluate it in the future
version.
> Release Testing: Verify FLINK-29663 Further improvements of adaptive batch
> scheduler
> ------------------------------------------------------------------------------------
>
> Key: FLINK-31079
> URL: https://issues.apache.org/jira/browse/FLINK-31079
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Lijie Wang
> Assignee: miamiaoxyz
> Priority: Blocker
> Fix For: 1.17.0
>
> Attachments: image-2023-02-22-14-00-13-646.png
>
>
> This task aims to verify FLINK-29663 which improves the adaptive batch
> scheduler.
> Before the change of FLINK-29663, adaptive batch scheduler will distribute
> subpartitoins according to the number of subpartitions, make different
> downstream subtasks consume roughly the same number of subpartitions. This
> will lead to imbalance loads of different downstream tasks when the
> subpartitions contain different amounts of data.
> To solve this problem, in FLINK-29663, we let the adaptive batch scheduler
> distribute subpartitoins according to the amount of data, so that different
> downstream subtasks consume roughly the same amount of data. Note that
> currently it only takes effect for All-To-All edges.
> The documentation of adaptive scheduler can be found
> [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler]
> One can verify it by creating intended data skew on All-To-All edges.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
