Does a Kafka Streams consumer also have that same limitation of possible duplicates?
Thanks, Chris On Fri, Sep 27, 2019 at 11:56 AM Matthias J. Sax <matth...@confluent.io> wrote: > Enabling "read_committed" only ensures that a consumer does not return > uncommitted data. > > However, on failure, a consumer might still read committed messages > multiple times (if you commit offsets after processing). If you commit > offsets before you process messages, and a failure happens before > processing finishes, you may "loose" those messages, as they won't be > consumed again on restart. > > Hence, if you have a "consumer only" application, not much changed and > you still need to take care in your application code about potential > duplicate processing of records. > > -Matthias > > > On 9/27/19 7:34 AM, Alessandro Tagliapietra wrote: > > You can achieve exactly once on a consumer by enabling read committed and > > manually committing the offset as soon as you receive a message. That way > > you know that at next poll you won't get old message again. > > > > On Fri, Sep 27, 2019, 6:24 AM christopher palm <cpa...@gmail.com> wrote: > > > >> I had a similar question, and just watched the video on the > confluent.io > >> site about this. > >> From what I understand idempotence and transactions are there to solve > the > >> duplicate writes and exactly once processing, respectively. > >> > >> Is what you are stating below is that this only works if we produce > into a > >> kafka topic and consume from it via a kafka stream, but a regular > >> kafka consumer won't get the guarantee of exactly once processing? > >> > >> Thanks, > >> Chris > >> > >> > >> On Sat, Aug 31, 2019 at 12:29 AM Matthias J. Sax <matth...@confluent.io > > > >> wrote: > >> > >>> Exactly-once on the producer will only ensure that no duplicate writes > >>> happen. If a downstream consumer fails, you might still read message > >>> multiple times for all cases (ie, without idempotence, with idempotence > >>> enabled, or if you use transactions). > >>> > >>> Note, that exactly-once is designed for a read-process-write pattern, > >>> but not for a write-read pattern. > >>> > >>> -Matthias > >>> > >>> > >>> > >>> On 8/30/19 1:00 PM, Peter Groesbeck wrote: > >>>> For a producer that emits messages to a single topic (i.e. no single > >>>> message is sent to multiple topics), will enabling idempotency but not > >>>> transactions provide exactly once guarantees for downstream consumers > >> of > >>>> said topic? > >>>> > >>>> Ordering is not important I just want to make sure consumers only > >>> consumer > >>>> messages sent once. > >>>> > >>> > >>> > >> > > > >
