[
https://issues.apache.org/jira/browse/FLINK-28884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-28884:
-------------------------------
Description:
Downstream task may never be notified of data available in hybrid shuffle when
number of credits is zero.
There are two potential problems:
# HsSubpartitionView should be initialized to a notifiable state, there may be
a problem of never consuming otherwise. Imagine the following situation:
Downstream task has no initial credit(i.e. exclusive buffers is configured to
zero), if there is no data output in the upstream, it will feedback a zero
backlog to downstream input channel. All subsequent data available
notifications will be intercepted as needNotify is false.
# HsSubpartitionView should be notifiable when downstream get a zero backlog.
Generally speaking, if the backlog is zero, when data become available, even if
there is no credit, the backlog information will be notified also. However, in
the hybrid shuffle, the notification will be ignored. This behavior is
incorrect.
was:
Downstream task may never be notified of data available in hybrid shuffle when
number of credits is zero.
There are two potential problems:
1. HsSubpartitionView should be initialized to a notifiable state, there may be
a problem of never consuming otherwise. Imagine the following situation:
Downstream task has no initial credit(i.e. exclusive buffers is configured to
zero), if there is no data output in the upstream, it will feedback a zero
backlog to downstream input channel. All subsequent data available
notifications will be intercepted as needNotify is false.
> Downstream task may never be notified of data available in hybrid shuffle
> when number of credits is zero.
> ---------------------------------------------------------------------------------------------------------
>
> Key: FLINK-28884
> URL: https://issues.apache.org/jira/browse/FLINK-28884
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.16.0
> Reporter: Weijie Guo
> Priority: Major
> Fix For: 1.16.0
>
>
> Downstream task may never be notified of data available in hybrid shuffle
> when number of credits is zero.
> There are two potential problems:
> # HsSubpartitionView should be initialized to a notifiable state, there may
> be a problem of never consuming otherwise. Imagine the following situation:
> Downstream task has no initial credit(i.e. exclusive buffers is configured to
> zero), if there is no data output in the upstream, it will feedback a zero
> backlog to downstream input channel. All subsequent data available
> notifications will be intercepted as needNotify is false.
> # HsSubpartitionView should be notifiable when downstream get a zero
> backlog. Generally speaking, if the backlog is zero, when data become
> available, even if there is no credit, the backlog information will be
> notified also. However, in the hybrid shuffle, the notification will be
> ignored. This behavior is incorrect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
