You need to do atomic actions with processing and saving the partition/offsets , while rebalance or assign or on initial start events you read the offset from the outside store, there are documentation and examples on the internet, what type of processing are you doing ?
בתאריך יום ו׳, 16 ביולי 2021, 19:01, מאת Pushkar Deole < pdeole2...@gmail.com>: > Chris, > > I am not sure how this solves the problem scenario that we are experiencing > in customer environment: the scenario is: > 1. application consumed record and processed it > 2. the processed record is produced on destination topic and ack is > received > 3. Before committing offset back to consumed topic, the application pod > crashed or shut down by kubernetes or shut down due to some other issue > > On Fri, Jul 16, 2021 at 8:57 PM Chris Larsen <clar...@confluent.io.invalid > > > wrote: > > > It is not possible out of the box, it is something you’ll have to write > > yourself. Would the following work? > > > > Consume -> Produce to primary topic-> get success ack back -> commit the > > consume > > > > Else if ack fails, produce to dead letter, then commit upon success > > > > Else if dead letter ack fails, exit (and thus don’t commit) > > > > Does that help? Someone please feel free to slap my hand but seems legit > to > > me ;) > > > > Chris > > > > > > > > On Fri, Jul 16, 2021 at 10:48 Pushkar Deole <pdeole2...@gmail.com> > wrote: > > > > > Thanks Chris for the response! > > > > > > The current application is quite evolved and currently using > > > > > > consumer-producer model described above and we need to fix some bugs > soon > > > > > > for a customer. So, moving to kafka streams seems bigger work. That's > why > > > > > > looking at work around if same thing can be achieved with current model > > > > > > using transactions that span across consumer offset commits and > producer > > > > > > send. > > > > > > > > > > > > We have made the producer idempotent and turned on transactions. > > > > > > However want to make offset commit to consumer and send from producer > to > > be > > > > > > atomic? Is that possible? > > > > > > > > > > > > On Fri, Jul 16, 2021 at 6:18 PM Chris Larsen > > <clar...@confluent.io.invalid > > > > > > > > > > wrote: > > > > > > > > > > > > > Pushkar, in kafka development for customer consumer/producer you > handle > > > it. > > > > > > > However you can ensure the process stops (or sends message to dead > > > letter) > > > > > > > before manually committing the consumer offset. On the produce side > you > > > can > > > > > > > turn on idempotence or transactions. But unless you are using > Streams, > > > you > > > > > > > chain those together yoursef. Would kafka streams work for the > > operation > > > > > > > you’re looking to do? > > > > > > > > > > > > > > Best, > > > > > > > Chris > > > > > > > > > > > > > > On Fri, Jul 16, 2021 at 08:30 Pushkar Deole <pdeole2...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am using a normal kafka consumer-producer in my microservice, > with > > a > > > > > > > > > > > > > > > > simple model of consume from source topic -> process the record -> > > > > > > > produce > > > > > > > > > > > > > > > > on destination topic. > > > > > > > > > > > > > > > > I am mainly looking for exactly-once guarantee wherein the offset > > > commit > > > > > > > > > > > > > > > > to consumed topic and produce on destination topic would both > happen > > > > > > > > > > > > > > > > atomically or none of them would happen. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In case of failures of service instance, if consumer has consumed, > > > > > > > > > > > > > > > > processed record and produced on destination topic but offset not > yet > > > > > > > > > > > > > > > > committed back to source topic then produce should also not happen > on > > > > > > > > > > > > > > > > destination topic. > > > > > > > > > > > > > > > > Is this behavior i.e. exactly-once, across consumers and producers, > > > > > > > > > > > > > > > > possible with transactional support in kafka? > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > [image: Confluent] <https://www.confluent.io> > > > > > > > Chris Larsen > > > > > > > Sr Solutions Engineer > > > > > > > +1 847 274 3735 <+1+847+274+3735> > > > > > > > Follow us: [image: Blog] > > > > > > > < > > > > > > > > > > > > > https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog > > > > > > > >[image: > > > > > > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > > > > > > <https://www.linkedin.com/in/chrislarsen/> > > > > > > > > > > > > > -- > > > > > > [image: Confluent] <https://www.confluent.io> > > Chris Larsen > > Sr Solutions Engineer > > +1 847 274 3735 <+1+847+274+3735> > > Follow us: [image: Blog] > > < > > > https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog > > >[image: > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > <https://www.linkedin.com/in/chrislarsen/> > > >
