[
https://issues.apache.org/jira/browse/FLINK-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhijiang updated FLINK-8547:
----------------------------
Description:
Currently in exactly-once mode, the {{BarrierBuffer}} would block inputs with
barriers until all inputs have received the barrier for a given checkpoint. To
avoid back-pressuring the input streams which may cause distributed deadlocks,
the {{BarrierBuffer}} has to spill the data in disk files to recycle the
buffers for blocked channels.
Based on credit-based flow control, every channel has exclusive buffers, so it
is no need to spill data for avoiding deadlock. Then we implement a new
{{CheckpointBarrierHandler}} for only buffering the data for blocked channels
for better performance.
And this new {{CheckpointBarrierHandler}} can also be configured to use or not
in order to rollback the original mode for unexpected risks.
was:
Currently in exactly-once mode, the `BarrierBuffer` would block inputs with
barriers until all inputs have received the barrier for a given checkpoint. To
avoid back-pressuring the input streams which may cause distributed deadlocks,
the `BarrierBuffer` has to spill the data in disk files to recycle the buffers
for blocked channels.
Based on credit-based flow control, every channel has exclusive buffers, so it
is no need to spill data for avoiding deadlock. Then we implement a new
'CheckpointBarrierHandler` for only buffering the data for blocked channels for
better performance.
And this new `CheckpointBarrierHandler` can also be configured to use or not in
order to rollback the original mode for unexpected risks.
> Implement CheckpointBarrierHandler not to spill data for exactly-once based
> on credit-based flow control
> --------------------------------------------------------------------------------------------------------
>
> Key: FLINK-8547
> URL: https://issues.apache.org/jira/browse/FLINK-8547
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.5.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
>
> Currently in exactly-once mode, the {{BarrierBuffer}} would block inputs with
> barriers until all inputs have received the barrier for a given checkpoint.
> To avoid back-pressuring the input streams which may cause distributed
> deadlocks, the {{BarrierBuffer}} has to spill the data in disk files to
> recycle the buffers for blocked channels.
>
> Based on credit-based flow control, every channel has exclusive buffers, so
> it is no need to spill data for avoiding deadlock. Then we implement a new
> {{CheckpointBarrierHandler}} for only buffering the data for blocked channels
> for better performance.
>
> And this new {{CheckpointBarrierHandler}} can also be configured to use or
> not in order to rollback the original mode for unexpected risks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
