Hi Sebastian, Hi Dawid, As part of this FLIP, the `AlignedSplitReader` interface (aka the stop & resume behavior) will be implemented for Kafka and Pulsar only, correct?
+1 in general. I believe it is valuable to complete the watermark aligned story with this FLIP. Cheers, Konstantin On Thu, Apr 21, 2022 at 12:36 PM Dawid Wysakowicz <dwysakow...@apache.org> wrote: > To be explicit, having worked on it, I support it ;) I think we can > start a vote thread soonish, as there are no concerns so far. > > Best, > > Dawid > > On 13/04/2022 11:27, Sebastian Mattheis wrote: > > Dear Flink developers, > > > > I would like to open a discussion on FLIP 217 [1] for an extension of > > Watermark Alignment to perform alignment also in SplitReaders. To do so, > > SplitReaders must be able to suspend and resume reading from split > sources > > where the SourceOperator coordinates and controlls suspend and resume. To > > gather information about current watermarks of the SplitReaders, we > extend > > the internal WatermarkOutputMulitplexer and report watermarks to the > > SourceOperator. > > > > There is a PoC for this FLIP [2], prototyped by Arvid Heise and revised > and > > reworked by Dawid Wysakowicz (He did most of the work.) and me. The > changes > > are backwards compatible in a way that if affected components do not > > support split alignment the behavior is as before. > > > > Best, > > Sebastian > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-217+Support+watermark+alignment+of+source+splits > > [2] https://github.com/dawidwys/flink/tree/aligned-splits > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
