[
https://issues.apache.org/jira/browse/FLINK-24189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz closed FLINK-24189.
------------------------------------
Assignee: Dawid Wysakowicz
Resolution: Implemented
Implemented in
6d9cca4de1c370b15a5f67b84efb413db2834ccf..f6d40bcfd89e67647d1f9cb41fa8be613d122494
> Buffer debloating for multiple gates
> ------------------------------------
>
> Key: FLINK-24189
> URL: https://issues.apache.org/jira/browse/FLINK-24189
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Anton Kalashnikov
> Assignee: Dawid Wysakowicz
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Right now, the buffer debloat assumes that it works with
> {{SingleInputGates}}, each of which has a similar load. The goal is to
> improve the {{UnionInputGate}} to behave well in case of data skew.
> A possible implementation can be to calculate the throughput separately for
> each gate. Another option is to calculate the total throughput but choose the
> buffer size independently for each gate based on their buffers in use.
> The base idea is to keep the same number of buffers in use for each gate. It
> means if a gate has a small number of buffers in use we decrease the buffer
> size to force this gate to use more buffers, and we move the withdrawn size
> to another gate with a greater number of buffers in use to increase its
> buffer size. In the corner case when all gates(or some gates) use all their
> buffers, the buffer size should be equal in each gate.
> It is highly important to fairly share the throughput among all gates. In
> other words, we need to avoid the situation:
> * gate1 has a low load while gate2 has a high load
> * the small buffer size was set for gate1 and the big buffer size for gate2
> * the load for gate1 increased up to the load of gate2
> * it is impossible to increase the buffer size for gate1 because there is no
> reason to decrease the buffer size for gate2 since the load for it doesn't
> change
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
