Yes, the StreamingFileSink is not affected.
Best,
Fabian
Thanks Fabian.
Do you know if the legacy StreamingFileSink has the same issues? Not asking
you to do research but just wondering if you happen to know.
On Tue, Nov 29, 2022 at 8:46 AM Fabian Paul wrote:
> Hi folks,
>
> I did some initial investigation, and the problem seems twofold.
>
> If no
Hi folks,
I did some initial investigation, and the problem seems twofold.
If no post-commit topology is used, we do not run into a problem where
we could lose data but since we do not clean up the state correctly,
we will hit this [1] when trying to stop the pipeline with a savepoint
after we
Let's loop in Fabian to clarify. I'm not sure if this only occurs when
using a post-commit topology (like compaction), but he can definitely
clarify :)
On Tue, Nov 29, 2022 at 2:19 PM Galen Warren
wrote:
> This seems scary -- am I interpreting it correctly to mean that unified
> FileSink
This seems scary -- am I interpreting it correctly to mean that unified
FileSink doesn't work properly with jobs that need to be
stopped-with-savepoints and restarted?
Should one use the deprecated StreamingFileSink until this is resolved?
On Tue, Nov 29, 2022 at 6:02 AM Fabian Paul (Jira)
Fabian Paul created FLINK-30238:
---
Summary: Unified Sink committer does not clean up state on final
savepoint
Key: FLINK-30238
URL: https://issues.apache.org/jira/browse/FLINK-30238
Project: Flink
