Hi Shekhar, Thanks for the KIP. Adding transactional support for acknowledgements is one of the future features I'd like to see for share groups, so I'm glad you've started this.
Here are some comments from an initial read. AS1: Previously, Kafka transactions have always been a producer concept. You set a configuration called `transactional.id` in the producer configuration which gives authorization checking and helps with fencing. Then you use KafkaProducer methods like beginTransaction and commitTransaction to mark the start and end of the transaction. Only the producer is currently enabled for transactional operation, and the Kafka RPCs it uses include transactional ID as necessary to indicate operations which are transactional. There will be differences for sure with share groups, but the overall model should probably still hold. Over the years, we've gradually tightened up the transaction protocol by adding features such as the producer epoch and two-phase commit. I would rather not re-invent everything. For example, WriteTxnMarkers is used by the transaction coordinator to tell the group coordinator to complete transactional operations on the __consumer_offsets topic. I would expect a similar kind of interaction with the share coordinator on the __share_group_state topic. I suggest looking at how to adapt the existing Kafka protocol RPCs rather than defining new ones. AS2: Why is there an acknowledge(ConsumerRecord<,>, AcknowledgeType, String) method? I would have expected that the transactional ID is a configuration, not a parameter of this method. AS3: Have you considered how to write an application which uses a share group and producer, and coordinates their operations in the same transaction? Thanks, Andrew On 2026/02/24 08:54:36 Shekhar Prasad Rajak via dev wrote: > Hello team, > In this discussion thread we will finalise the design for the KIP-1289 > Support Hierarchical Transactional Acknowledgments for Share Groups. > > Share Groups currently support only immediate acknowledgement modes > (IMPLICIT/EXPLICIT). This causes data loss in distributed streaming > frameworks: > > 1. Worker acknowledges records → Records removed from Kafka > 2. Checkpoint fails before sink write > 3. Records lost (acknowledged but never persisted) > > Goal: Enable exactly-once read semantics via transactional acknowledgements. > We already have similar pattern in Kafka producer transaction > > KIP: KIP-1289 Support Hierarchical Transactional Acknowledgments for Share > Groups - Apache Kafka - Apache Software Foundation > > Looking forward to community's feedback! > > > Regards,Shekhar Prasad Rajak > >
