[
https://issues.apache.org/jira/browse/FLINK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518862#comment-17518862
]
Anton Kalashnikov commented on FLINK-26762:
-------------------------------------------
[~fanrui], Thanks for your proposal. It looks valid. As I understand, the main
benefit is to wait for an available buffer outside the operator which indeed
allows starting checkpoint while there are no available buffers. So it makes
sense. I just want to share some of my concerns/notices about this:
* We need to formulate more specifically what is the target case for this
improvement. For example, you said about a large record but in fact, I don't
think it is valid since we already reset to unavailability when at least one
buffer is available yet. This means that this large record should be greater
than the size of the segment but in this case, we strongly recommend increasing
the segment size to avoid such a situation. at the same time `flatmap` is valid
case I think, what else? window operator?
* I am also a little concerned about the new configuration. Since we already
have a lot of buffer configurations(exclusive/floating buffers number and size,
max buffers per channel) which are not so easy to setup correctly.
Unfortunately, I don't see any other good solution right now. But I just want
emphasize that we should be careful with new configuration since it is not
always easy for understanding.
* It is also a good idea to understand how helpful is this improvement. So
perhaps, it makes sense to have a benchmark for it or use some existing
benchmark. But any case, we should somehow understand that it works as we
expected.
* I don't think that it will predictably work with buffer debloating. Maybe it
is a not so big problem but anyway we should take it into account.
One more idea for future improvement, we can adjust the number of overdraft
buffers or even overdraft buffer size automatically as we do it for buffer
debloating now. For example, we can calculate median/average operator output
size and always keep the same size of memory available. Perhaps in this case we
don't even need configuration for exact number of overdraft buffers (but we
highly likely need another configuration for managing this algorithm).
P. S. [~fanrui], do you want to work on this task yourself?
> Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being
> blocked
> -----------------------------------------------------------------------------------
>
> Key: FLINK-26762
> URL: https://issues.apache.org/jira/browse/FLINK-26762
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network
> Affects Versions: 1.13.0, 1.14.0, 1.15.0
> Reporter: fanrui
> Priority: Major
> Fix For: 1.16.0
>
>
> In some past JIRAs of Unaligned Checkpoint, the community has added the
> recordWriter.isAvaliable() to reduce block for single record write. But for
> large record, flatmap or broadcast watermark, they may need more buffer.
> Can we add the overdraft buffer in BufferPool to reduce unaligned checkpoint
> being blocked?
> h2. Overdraft Buffer mechanism
> Add the configuration of
> 'taskmanager.network.memory.overdraft-buffers-per-gate=5'.
> When requestMemory is called and the bufferPool is insufficient, the
> bufferPool will allow the Task to overdraw up to 5 MemorySegments. And
> bufferPool will be unavailable until all overdrawn buffers are consumed by
> downstream tasks. Then the task will wait for bufferPool being available.
> From the above, we have the following benefits:
> * For scenarios that require multiple buffers, the Task releases the
> Checkpoint lock, so the Unaligned Checkpoint can be completed quickly.
> * We can control the memory usage to prevent memory leak.
> * It just needs a litter memory, and can improve the stability of the Task
> under back pressure.
> * Users can increase the overdraft-buffers to adapt the scenarios that
> require more buffers.
>
> Masters, please correct me if I'm wrong, thanks a lot.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
