Hi, Ajo, Sorry to reply this late. Could you clarify one thing in the design: For watermark triggered window draining, is the infinitive watermark trigger happen first, or the drain token in all source SSP happen first? Shouldn't it be the following sequence: a) all drain token from all input source SSPs (except for intermediate streams) are received by tasks ==> b) infinite watermark triggers from the source and flush all window/triggers in the pipeline ==> c) once the infinite watermark is propagated through all stages in the pipeline, stops the tasks. Could you confirm?
Thanks a lot! -Yi On Thu, Nov 17, 2022 at 9:48 AM Ajo Thomas <ajo.thoma...@gmail.com> wrote: > Hi All, > > Samza currently doesn't have a way to gracefully drain pipelines before > making a backward-incompatible intermediate schema change. We have added a > feature called Pipeline Drain to the samza engine to address this problem. > Here is the SEP page for it: > > https://cwiki.apache.org/confluence/display/SAMZA/SEP-31%3A+Pipeline+Drain%3A+Support+the+ability+to+drain+pipelines+to+allow+incompatible+intermediate+schema+changes > > > If there are no major blockers, we are tentatively seeking to open a vote > on Monday, Nov 28th, 2022. > > Thanks, > Ajo >
