[
https://issues.apache.org/jira/browse/FLINK-30189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-30189:
-----------------------------------
Labels: pull-request-available (was: )
> HsSubpartitionFileReader may load data that has been consumed from memory
> -------------------------------------------------------------------------
>
> Key: FLINK-30189
> URL: https://issues.apache.org/jira/browse/FLINK-30189
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.16.0, 1.17.0
> Reporter: Weijie Guo
> Assignee: Weijie Guo
> Priority: Major
> Labels: pull-request-available
>
> In order to solve the problem that data cannot be read from the disk
> correctly after failover, we changed the calculation logical of the buffer's
> readable state in FLINK-29238. Buffers that are greater than consumingOffset
> and have been released can be pre-load from file. However, the update of
> consumingOffset is asynchronous, If it lags behind the actual consumption
> progress, the buffer will have a chance to be load from the disk again.
> IMO, we can record the consumed status of buffer by each consumer in the
> InternalRegion. Only the buffers that have not been consumed and have been
> released will be considered as readable. In the case of failover, a new
> consumerId will be generated, so all buffers will be considered as unconsumed
> and can be correctly read from the disk too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
