Hi Lerh Chuan Low, MAny thanks for your response. I get it now, that it provides exactly-once semantics i.e it looks to user that it is processed exactly once. Also, i am clear on the aspect about read_committed level so the uncommitted transaction and hence uncommitted send won't be visible to consumers.
However one last query i have is how to make sure that as part of the same transaction, i am also sending and also committing offsets. Which API should i look at: is this correct API : KafkaProducer. sendOffsetsToTransaction On Fri, Jul 16, 2021 at 9:57 PM Lerh Chuan Low <lerhch...@gmail.com> wrote: > Pushkar, > > My understanding is you can easily turn it on by using Kafka streams as > Chris mentioned. Otherwise you'd have to do it yourself - I don't think you > can get exactly once processing, but what you can do (which is also what > Kafka streams does) is exactly once schematics (You won't be able to get > every message processed exactly once in the system, but they could look > like they had been processed exactly once), The final piece of the puzzle > besides using idempotent producers and transactions is to set consumers of > the downstream topic to *read_committed: true*. So in your example the > messages would still have made it to the destination topic, however because > the transaction has not yet been completed, the downstream consumer would > ignore them. > > You can still only do exactly once processing up to the boundaries of > Kafka, that said. Wherever Kafka terminates you'd have to code it yourself. > > > > On Sat, Jul 17, 2021 at 2:01 AM Pushkar Deole <pdeole2...@gmail.com> > wrote: > > > Chris, > > > > I am not sure how this solves the problem scenario that we are > experiencing > > in customer environment: the scenario is: > > 1. application consumed record and processed it > > 2. the processed record is produced on destination topic and ack is > > received > > 3. Before committing offset back to consumed topic, the application pod > > crashed or shut down by kubernetes or shut down due to some other issue > > > > On Fri, Jul 16, 2021 at 8:57 PM Chris Larsen > <clar...@confluent.io.invalid > > > > > wrote: > > > > > It is not possible out of the box, it is something you’ll have to write > > > yourself. Would the following work? > > > > > > Consume -> Produce to primary topic-> get success ack back -> commit > the > > > consume > > > > > > Else if ack fails, produce to dead letter, then commit upon success > > > > > > Else if dead letter ack fails, exit (and thus don’t commit) > > > > > > Does that help? Someone please feel free to slap my hand but seems > legit > > to > > > me ;) > > > > > > Chris > > > > > > > > > > > > On Fri, Jul 16, 2021 at 10:48 Pushkar Deole <pdeole2...@gmail.com> > > wrote: > > > > > > > Thanks Chris for the response! > > > > > > > > The current application is quite evolved and currently using > > > > > > > > consumer-producer model described above and we need to fix some bugs > > soon > > > > > > > > for a customer. So, moving to kafka streams seems bigger work. That's > > why > > > > > > > > looking at work around if same thing can be achieved with current > model > > > > > > > > using transactions that span across consumer offset commits and > > producer > > > > > > > > send. > > > > > > > > > > > > > > > > We have made the producer idempotent and turned on transactions. > > > > > > > > However want to make offset commit to consumer and send from producer > > to > > > be > > > > > > > > atomic? Is that possible? > > > > > > > > > > > > > > > > On Fri, Jul 16, 2021 at 6:18 PM Chris Larsen > > > <clar...@confluent.io.invalid > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Pushkar, in kafka development for customer consumer/producer you > > handle > > > > it. > > > > > > > > > However you can ensure the process stops (or sends message to dead > > > > letter) > > > > > > > > > before manually committing the consumer offset. On the produce side > > you > > > > can > > > > > > > > > turn on idempotence or transactions. But unless you are using > > Streams, > > > > you > > > > > > > > > chain those together yoursef. Would kafka streams work for the > > > operation > > > > > > > > > you’re looking to do? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > On Fri, Jul 16, 2021 at 08:30 Pushkar Deole <pdeole2...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am using a normal kafka consumer-producer in my microservice, > > with > > > a > > > > > > > > > > > > > > > > > > > > simple model of consume from source topic -> process the record > -> > > > > > > > > > produce > > > > > > > > > > > > > > > > > > > > on destination topic. > > > > > > > > > > > > > > > > > > > > I am mainly looking for exactly-once guarantee wherein the > offset > > > > commit > > > > > > > > > > > > > > > > > > > > to consumed topic and produce on destination topic would both > > happen > > > > > > > > > > > > > > > > > > > > atomically or none of them would happen. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In case of failures of service instance, if consumer has > consumed, > > > > > > > > > > > > > > > > > > > > processed record and produced on destination topic but offset not > > yet > > > > > > > > > > > > > > > > > > > > committed back to source topic then produce should also not > happen > > on > > > > > > > > > > > > > > > > > > > > destination topic. > > > > > > > > > > > > > > > > > > > > Is this behavior i.e. exactly-once, across consumers and > producers, > > > > > > > > > > > > > > > > > > > > possible with transactional support in kafka? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > [image: Confluent] <https://www.confluent.io> > > > > > > > > > Chris Larsen > > > > > > > > > Sr Solutions Engineer > > > > > > > > > +1 847 274 3735 <+1+847+274+3735> > > > > > > > > > Follow us: [image: Blog] > > > > > > > > > < > > > > > > > > > > > > > > > > > > > https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog > > > > > > > > > >[image: > > > > > > > > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > > > > > > > > <https://www.linkedin.com/in/chrislarsen/> > > > > > > > > > > > > > > > > > -- > > > > > > > > > [image: Confluent] <https://www.confluent.io> > > > Chris Larsen > > > Sr Solutions Engineer > > > +1 847 274 3735 <+1+847+274+3735> > > > Follow us: [image: Blog] > > > < > > > > > > https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog > > > >[image: > > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > > <https://www.linkedin.com/in/chrislarsen/> > > > > > >
