How easy is the migration to the StreamingFileSink?
On 10/13/2020 1:01 PM, Aljoscha Krettek wrote:
On 13.10.20 11:18, David Anderson wrote:
I think the pertinent question is whether there are interesting cases
where
the BucketingSink is still a better choice. One case I'm not sure
about is
the situation described in docs for the StreamingFileSink under
Important
Note 2 [1]:
... upon normal termination of a job, the last in-progress files
will
not be transitioned to the “finished” state.
I know this confuses and frustrates users, but I don't know if the
BucketingSink has any advantages in this regard.
The BucketingSink suffers from the same problem. It's caused by the
fact that we don't do a "final" checkpoint before shutting down a
pipeline. We're trying to resolve that with FLIP-147 [1].
[1] https://cwiki.apache.org/confluence/x/mw-ZCQ
