[
https://issues.apache.org/jira/browse/FLINK-11037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714407#comment-16714407
]
zhijiang commented on FLINK-11037:
----------------------------------
[~till.rohrmann]， thanks for paying attention for this issue! :)
* The easy and extreme way is try to take all based on sender's backlog. A
more gentle way is giving a proper batch of buffers based on how many buffers
available and how many channels altogether on {{LocalBufferPool}} side.
* Yes, it will introduce another method for requesting buffers in batch just
as you provided.
* You pointed out a very good and critical question. I think the precondition
is how to measure the performance for different algorithms. I would like to add
some further factors like metrics to monitor whether the current credit-based
is smooth enough, and then we can judge which algorithm behaves better in
different scenarios. It does not suggest the user choosing the algorithm
otherwise we can give specific instructions of how to tune the algorithm for
different scenarios based on our reliable experiments. And it may be better to
adjust the algorithm automatically based on some hints or real-time metrics.
> Introduce another greedy mechanism for distributing floating buffers
> --------------------------------------------------------------------
>
> Key: FLINK-11037
> URL: https://issues.apache.org/jira/browse/FLINK-11037
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.8.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> The current mechanism for distributing floating buffers is fair for all the
> listeners. In detail, each input channel can only request one floating buffer
> each time although this channel may actually need more floating buffers. Then
> this channel has to loop to request floating buffer until all are satisfied
> or pool is exhausted.
> In generally speaking, this way seems fair for all the concurrent channels
> invoked by netty nio thread. But every request from LocalBufferPool needs to
> syn lock and it is hard to say how to distribute all the available floating
> buffers behaves better in real scenarios.
> Therefore we propose another greedy mechanism to request more floating
> buffers each time. In extreme case, we can even request all the required
> buffers at a time or partial ones via configured parameters. On the other
> side, LocalBufferPool can also decide how many floating buffers should been
> assigned based on some factors, such as how many total channels and how many
> total floating buffers.
> The motivation is making better use of floating buffer resources and it may
> need extra metrics for adjusting the mechanism dynamically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
