Not exactly sure what you mean by "restored"? In general, there are two types of punctuation: wall-clock time and event-time.
For wall-clock time, it's somewhat non-deterministic when they will be called. For event-time, as those depend on the immutable input, they will be called again as in the failure run, when event/stream time progress accordingly. Hope this answers your question. -Matthias On 9/21/18 11:22 PM, Vishnu Viswanath wrote: > Thank you Matthias, > > That cleared things up, have one follow up question on punctuations. > > Does punctuations get restored after failure? If not what is the right way > to do so? Is it by storing the info currently active punctuations in some > state and by scheduling them again in the init method? > > Thanks, > Vishnu > > On Thu, Sep 20, 2018 at 3:42 PM Matthias J. Sax <[email protected]> > wrote: > >> 1. The record will be re-read, but the state would not be re-build (ie, >> no undo of step (2). Thus, on re-processing you would add the record >> again, and you would "over count" in step (3) -- trigger would still >> fire I assume. >> >> 2. I assume, by "forward" you mean writing to an output topic: If you >> write to an output topic, the Producer will buffer multiple records to >> increase write efficiency. Thus, if can happen that the write is done >> immediately (if the buffer is full) or later. So, yes, there could be >> duplicate in the output topic, because the write can happen before flush. >> >> 3. As mentioned above, the write can happen before flushing >> >> 4. Yes. For this case, the state would be reset, too. >> >> >> -Matthias >> >> On 9/18/18 10:09 PM, Vishnu Viswanath wrote: >>> Hi All, >>> >>> I have KafkaStreams application (processor API) that does roughly the >> below >>> steps. >>> >>> 1. read >>> 2. add to state >>> 3. check state size (count based trigger) >>> 3.1 process >>> 3.2 delete records from state >>> 3.3 forward >>> 3.4 commit >>> (kafka internally does) >>> 3.4.1 flush state >>> 3.4.2 flush producer >>> 3.4.3 commit offset >>> >>> Have the following questions regarding failures at different steps: >>> >>> 1. Am I correct in assuming that failure at steps 1, 2, 3.1 & 3.2 is fine >>> (will read records from last committed offset and rebuild the state and >>> continue with rest of the steps. >>> 2. When will forward at step 3.3 have effect? will it be sent out only at >>> 3.4.2 (when producer is flushed) or can it happen before that? if it can >>> happen before, then any failure after 3.3 can cause duplicates? >>> 3. if forward will take effect only during flush, still a failure between >>> 3.4.2 and 3.4.3 can cause duplicates? >>> 4. willl setting processing.guarantee=exactly_once solve the duplicates >>> issue? >>> >>> Thanks, >>> Vishnu >>> >> >> >
signature.asc
Description: OpenPGP digital signature
