Thanks for preparing the FLIP, Xuannan. +1 in general.
A quick question, could you explain why we are relying on the watermark for emitting the record attribute? Why not use timestamps in the records? I don't see any concern in using watermarks. Just wondering if there's any deep considerations behind this. Best, Xintong On Thu, Aug 3, 2023 at 3:03 PM Xuannan Su <suxuanna...@gmail.com> wrote: > Hi all, > > I am opening this thread to discuss FLIP-328: Allow source operators to > determine isProcessingBacklog based on watermark lag[1]. We had a several > discussions with Dong Ling about the design, and thanks for all the > valuable advice. > > The FLIP aims to target the use-case where user want to run a Flink job to > backfill historical data in a high throughput manner and continue > processing real-time data with low latency. Building upon the backlog > concept introduced in FLIP-309[2], this proposal enables sources to report > their status of processing backlog based on the watermark lag. > > We would greatly appreciate any comments or feedback you may have on this > proposal. > > Best, > Xuannan > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-328%3A+Allow+source+operators+to+determine+isProcessingBacklog+based+on+watermark+lag > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog >
