Jiayi Liao created FLINK-22887:
----------------------------------
Summary: Backlog based optimizations for RebalancePartitioner and
RescalePartitioner
Key: FLINK-22887
URL: https://issues.apache.org/jira/browse/FLINK-22887
Project: Flink
Issue Type: Improvement
Components: Runtime / Network
Affects Versions: 1.13.1
Reporter: Jiayi Liao
{\{RebalancePartitioner}} uses round-robin to distribute the records but this
may not work as expected, because the environments and the processing ability
of the downstream tasks may differ from each other. In such cases, the
throughput of the whole job will be limited by the slowest downstream subtask,
which is very similar with the "HASH" scenario.
Instead, after the credit-based mechanism is introduced, we can leverage the
{{backlog}} on the sender side to identify the "load" on each receiver side,
which help us distribute the data more fairly in {{RebalancePartitioner}} and
{{RescalePartitioner}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
