[
https://issues.apache.org/jira/browse/FLINK-29663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang updated FLINK-29663:
-------------------------------
Description:
In Flink 1.15, we introduced the adaptive batch scheduler to automatically
decide parallelisms of job vertices for batch jobs. In this issue, we will
further optimize it by changing the subpartition range division algorithm:
change it from dividing according to the number of subpartitions(the number of
subpartitions within each subpartition range is basically the same) to dividing
according to the amount of data in subpartition ranges (the amount of data
within each subpartition range is basically the same).
More details see
[https://docs.google.com/document/d/1Qyq3qDkBCUNupajVJpFTKp3fHQJtwIu9luM7T52k1Oo]
This is the umbrella ticket for the improvements.
was:
In Flink 1.15, we introduced the adaptive batch scheduler to automatically
decide parallelisms of job vertices for batch jobs. In this issue, we will
further optimize it by changing the subpartition range division algorithm:
change it from dividing according to the number of subpartitions(the number of
subpartitions within each subpartition range is basically the same) to dividing
according to the amount of data in subpartition ranges (the amount of data
within each subpartition range is basically the same).
More details see
[https://docs.google.com/document/d/1Qyq3qDkBCUNupajVJpFTKp3fHQJtwIu9luM7T52k1Oo]
This is the umbrella ticket for the improvements.
> Further improvements of adaptive batch scheduler
> ------------------------------------------------
>
> Key: FLINK-29663
> URL: https://issues.apache.org/jira/browse/FLINK-29663
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Reporter: Lijie Wang
> Assignee: Lijie Wang
> Priority: Major
> Fix For: 1.17.0
>
>
> In Flink 1.15, we introduced the adaptive batch scheduler to automatically
> decide parallelisms of job vertices for batch jobs. In this issue, we will
> further optimize it by changing the subpartition range division algorithm:
> change it from dividing according to the number of subpartitions(the number
> of subpartitions within each subpartition range is basically the same) to
> dividing according to the amount of data in subpartition ranges (the amount
> of data within each subpartition range is basically the same).
> More details see
> [https://docs.google.com/document/d/1Qyq3qDkBCUNupajVJpFTKp3fHQJtwIu9luM7T52k1Oo]
> This is the umbrella ticket for the improvements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
