[
https://issues.apache.org/jira/browse/FLINK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108946#comment-18108946
]
Li Guo edited comment on FLINK-26755 at 8/27/26 6:40 PM:
---------------------------------------------------------
The cause is in FileWriterBucket#restoreInProgressFile: when it resumes the
in-progress file it never registers the restored recoverable for cleanup. The
legacy StreamingFileSink had the same bug and got fixed in FLINK-26151, the new
FileSink just never got that fix. FLINK-38842 (orphaned temp files in COS after
restoring) looks like the same thing in the wild.
I wrote a test in FileWriterBucketTest showing no cleanup committable is
emitted after restore-with-resume, and the fix is basically the same one-liner
as the legacy one.
This ticket has been open for a few years now, so a sanity check first: is this
still considered worth fixing? The bug is still on master and FLINK-38842
suggests people keep hitting it, so I'd say yes. If a committer agrees, could
you assign it to me? Patch is ready.
was (Author: liguo):
The cause is in FileWriterBucket#restoreInProgressFile: when it resumes the
in-progress file it never registers the restored recoverable for cleanup. The
legacy StreamingFileSink had the same bug and got fixed in FLINK-26151, the new
FileSink just never got that fix. FLINK-38842 (orphaned temp files in COS after
restoring) looks like the same thing in the wild.
I wrote a test in FileWriterBucketTest showing no cleanup committable is
emitted after restore-with-resume, and the fix is basically the same one-liner
as the legacy one.
This ticket has been open for a few years now, so a sanity check first: is this
still considered worth fixing? The bug is still on master and FLINK-38842
suggests people keep hitting it, so I'd say yes. If a committer agrees, could
you assign it to me? Patch is ready. [~guoxuanlin] you wrote on FLINK-38842
that you wanted to fix this, still working on it? Happy to defer if so.
> Cleanup inprogressfileRecoverable for FileSink on restoring
> -----------------------------------------------------------
>
> Key: FLINK-26755
> URL: https://issues.apache.org/jira/browse/FLINK-26755
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.15.0, 1.16.0
> Reporter: Yun Gao
> Priority: Major
>
> The FileSink has the similar issue to
> https://issues.apache.org/jira/browse/FLINK-26151
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
