[
https://issues.apache.org/jira/browse/FLINK-24191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412421#comment-17412421
]
Piotr Nowojski commented on FLINK-24191:
----------------------------------------
My best guess would be when debloating, first keep the number of buffers
unchanged, so that buffer transfer is fluent (1 buffer being written to, other
in transfer). So first start shrinking buffer size, until we reach some
threshold (1 or 2 records per buffer), then start shrinking number of buffers.
When throughput is increasing and we need more in-flight data keep the same
priority. Better to have a smaller but desired/configured amount of buffers,
rather than larger buffers.
Keep in mind that we are talking about scenarios when processing single record
takes very long time, so it shouldn't matter whether single buffer has 10
records or 5 records from the perspective of the throughput.
> Adjusting number of buffers besides buffer size
> -----------------------------------------------
>
> Key: FLINK-24191
> URL: https://issues.apache.org/jira/browse/FLINK-24191
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.14.0
> Reporter: Anton Kalashnikov
> Priority: Major
> Fix For: 1.15.0
>
>
> "Buffer debloat" adjusts only the buffer size but it also makes sense to
> adjust the number of buffers. It is not clear for now what should be adjusted
> and in which proportions so it needs to think about how to figure this out.
> The main idea of this ticket is to understand what is better to have one
> buffer of 10 records or 10 buffers of 1 record. On one hand handling of each
> buffer has an overhead on another hand, in case of low load it doesn't
> really make sense and it is ok to have many buffers with small sizes.
> Perhaps, we need benchmarks(microbenchamrk) to understand correlation between
> performance / buffers number / buffer size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
