[
https://issues.apache.org/jira/browse/FLINK-24191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412738#comment-17412738
]
Anton Kalashnikov commented on FLINK-24191:
-------------------------------------------
[~pnowojski], As I understand, the current idea is:
* BufferDebloater collects the information about record size in order to
calculate the average record size.
* if the desired buffer size is calculated less than the average record size we
keep the buffer size unchanged but we change the number of buffers.
newBufferCount = buffersInUse * desiredBufferSize / actualBufferSize
* newBufferCount is the total number of buffers that can be used in one gate by
all channels. (somehow it should be balanced between exclusive and floating
buffers)
* The number of buffers for the subpartition can be calculated by dividing
newBufferCount on the number of `channels with data` and this number should be
sent to all subpartitions regardless they have data right now or not.
> Adjusting number of buffers besides buffer size
> -----------------------------------------------
>
> Key: FLINK-24191
> URL: https://issues.apache.org/jira/browse/FLINK-24191
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.14.0
> Reporter: Anton Kalashnikov
> Priority: Major
> Fix For: 1.15.0
>
>
> "Buffer debloat" adjusts only the buffer size but it also makes sense to
> adjust the number of buffers. It is not clear for now what should be adjusted
> and in which proportions so it needs to think about how to figure this out.
> The main idea of this ticket is to understand what is better to have one
> buffer of 10 records or 10 buffers of 1 record. On one hand handling of each
> buffer has an overhead on another hand, in case of low load it doesn't
> really make sense and it is ok to have many buffers with small sizes.
> Perhaps, we need benchmarks(microbenchamrk) to understand correlation between
> performance / buffers number / buffer size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
