Thanks for working on this! I wonder if "supporting" split alignment in SourceReaderBase and then doing nothing if the split reader does not implement AlignedSplitReader could be misleading? Perhaps WithSplitsAlignment can instead be added to the specific source reader (i.e. KafkaSourceReader) to make it explicit that the source actually supports it.
Thanks, Thomas On Thu, Apr 21, 2022 at 4:57 AM Konstantin Knauf <kna...@apache.org> wrote: > Hi Sebastian, Hi Dawid, > > As part of this FLIP, the `AlignedSplitReader` interface (aka the stop & > resume behavior) will be implemented for Kafka and Pulsar only, correct? > > +1 in general. I believe it is valuable to complete the watermark aligned > story with this FLIP. > > Cheers, > > Konstantin > > > > > > > > On Thu, Apr 21, 2022 at 12:36 PM Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > > > To be explicit, having worked on it, I support it ;) I think we can > > start a vote thread soonish, as there are no concerns so far. > > > > Best, > > > > Dawid > > > > On 13/04/2022 11:27, Sebastian Mattheis wrote: > > > Dear Flink developers, > > > > > > I would like to open a discussion on FLIP 217 [1] for an extension of > > > Watermark Alignment to perform alignment also in SplitReaders. To do > so, > > > SplitReaders must be able to suspend and resume reading from split > > sources > > > where the SourceOperator coordinates and controlls suspend and resume. > To > > > gather information about current watermarks of the SplitReaders, we > > extend > > > the internal WatermarkOutputMulitplexer and report watermarks to the > > > SourceOperator. > > > > > > There is a PoC for this FLIP [2], prototyped by Arvid Heise and revised > > and > > > reworked by Dawid Wysakowicz (He did most of the work.) and me. The > > changes > > > are backwards compatible in a way that if affected components do not > > > support split alignment the behavior is as before. > > > > > > Best, > > > Sebastian > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-217+Support+watermark+alignment+of+source+splits > > > [2] https://github.com/dawidwys/flink/tree/aligned-splits > > > > > > > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
