Let's loop in Fabian to clarify. I'm not sure if this only occurs when using a post-commit topology (like compaction), but he can definitely clarify :)
On Tue, Nov 29, 2022 at 2:19 PM Galen Warren <ga...@cvillewarrens.com> wrote: > This seems scary -- am I interpreting it correctly to mean that unified > FileSink doesn't work properly with jobs that need to be > stopped-with-savepoints and restarted? > > Should one use the deprecated StreamingFileSink until this is resolved? > > On Tue, Nov 29, 2022 at 6:02 AM Fabian Paul (Jira) <j...@apache.org> > wrote: > > > Fabian Paul created FLINK-30238: > > ----------------------------------- > > > > Summary: Unified Sink committer does not clean up state on > > final savepoint > > Key: FLINK-30238 > > URL: https://issues.apache.org/jira/browse/FLINK-30238 > > Project: Flink > > Issue Type: Bug > > Components: Connectors / Common > > Affects Versions: 1.15.3, 1.17.0, 1.16.1 > > Reporter: Fabian Paul > > > > > > During stop-with-savepoint the committer only commits the pending > > committables on notifyCheckpointComplete. > > > > This has several downsides. > > * Last committableSummary has checkpoint id LONG.MAX and is never > cleared > > from the state leading to that stop-with-savepoint does not work when the > > pipeline recovers from a savepoint > > * While the committables are committed during stop-with-savepoint they > > are not forwarded to post-commit topology, potentially losing data and > > preventing to close open transactions. > > > > > > > > -- > > This message was sent by Atlassian Jira > > (v8.20.10#820010) > > >
