[
https://issues.apache.org/jira/browse/FLINK-30792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682353#comment-17682353
]
Feifan Wang commented on FLINK-30792:
-------------------------------------
Hi [~masteryhx] , you are right, but my suggestion is to delete the state
changes before the materialization trigger in notuploaded after the
materialization is completed. Otherwise this part of the state change will be
uploaded to the same file with subsequent state changes when triggering an
pre-emptive upload (with {_}BatchingStateChangeUploadScheduler{_}) . This is
one of the causes of changelog file not found.
>From the implementation of FsStateChangelogWriter and
>BatchingStateChangeUploadScheduler, there should be another situation that
>will cause the changelog file to not be found.
I will try to write test cases to reproduce these two cases.
> clean up not uploaded state changes after materialization complete
> ------------------------------------------------------------------
>
> Key: FLINK-30792
> URL: https://issues.apache.org/jira/browse/FLINK-30792
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.16.0
> Reporter: Feifan Wang
> Priority: Major
>
> We should clean up not uploaded state changes after materialization
> completed, otherwise it may cause FileNotFoundException.
> Since state changes before completed materialization in
> FsStateChangelogWriter#notUploaded will not be used in any subsequent
> checkpoint, I suggest clean up it while handle materialization result.
> How do you think about this ? [~ym] , [~roman]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
