Hi everyone, First off, thanks to the authors for the Feb 12th updates to KIP-1163 . Adding the periodic reconciliation loop clears up my concerns about the orphaned "Upload-then-Commit" segments, so I'm officially withdrawing my objection on the storage leak issue .
Chris and Greg- since you both mentioned digging into the 1164 details, I wanted to pick your brains on how Exactly-Once Semantics (EOS) is going to safely operate here. In standard Kafka, the Partition Leader is our single serialization point. It receives the data, tracks ongoing transactions via the ProducerStateManager, and calculates the Last Stable Offset (LSO) locally . Since KIP-1150 removes the leader, the Batch Coordinator takes over. But as I read through the current text, a few critical synchronization barriers seem to be missing to me: 1. LSO Calculation: How exactly will the Batch Coordinator maintain and calculate the LSO? Justine Olshan brought this up earlier too . Will the coordinator run its own ProducerStateManager to track ongoing transactions, or is there a totally different state machine planned? 2. RPC Protocol: What's the exact synchronization protocol between the legacy Transaction Coordinator and the new Batch Coordinator? When the Txn Coordinator sends a commit marker, how does the Batch Coordinator actually verify it has received all the prerequisite data batches for that specific transaction epoch? 3. Delayed Data Race Condition: Let's say a broker hits a GC pause right *after *uploading a batch to object storage, but *before* committing the coordinates . If the transaction commit marker arrives at the Coordinator first, what happens? Does the Coordinator wait? If not, couldn't the transaction commit with missing data, completely violating read_committed isolation? The KIP vaguely mentions *transactional checks* but leaves the actual commit protocol and public interfaces undefined right now . I'm not saying the design itself is broken, but I really think myself and others need to see these RPC flows explicitly documented before we implement and adopt this. Otherwise, we risk baking in some severe data isolation headaches down the line. Let me know what you guys think about locking down the text for these interfaces. Regards, Vaquar Khan On Wed, 25 Feb 2026 at 10:33, Greg Harris via dev <[email protected]> wrote: > Hey all, > > I'm excited to discuss more details in 1163 and 1164 with everyone. > > +1 (binding) > > Thanks! > Greg > > On Wed, Feb 25, 2026 at 1:08 AM Anatolii Popov via dev < > [email protected]> > wrote: > > > Hi all, > > > > Given the importance of this KIP, we want to keep the vote open for a few > > more days to give time to people who had comments in the DISCUSS thread > to > > cast their vote if they want. > > > > On Wed, Feb 25, 2026 at 10:47 AM Josep Prat via dev < > [email protected]> > > wrote: > > > > > Hi all, > > > As a co-author of the KIP, I want to explicitly cast my vote for this > > KIP. > > > > > > +1 (binding) > > > > > > > > > On Wed, Feb 25, 2026 at 9:02 AM Luke Chen <[email protected]> wrote: > > > > > > > I've re-read KIP-1150, and still agree this is what we need for > Apache > > > > Kafka. > > > > > > > > +1 (binding) from me. > > > > > > > > Thank you, > > > > Luke > > > > > > > > On Wed, Feb 25, 2026 at 12:10 PM Chris Egerton < > > [email protected]> > > > > wrote: > > > > > > > >> Hi all, > > > >> > > > >> Thanks for the KIP. I've reviewed 1150, 1163, and 1164, as well as > the > > > >> relevant discussion threads. I may have granular comments about 1163 > > and > > > >> 1164 but the overall approach suggested in 1150 looks good to me. I > > > >> especially like that the approach covers two main pain points of > > > operating > > > >> and paying for Kafka today: it allows cross-AZ traffic to be reduced > > > (even > > > >> eliminated in some cases), and it also allows local disk usage by > > > brokers > > > >> to be reduced (if operators opt for a small local cache on follower > > > >> brokers > > > >> for non-tiered segments). > > > >> > > > >> +1 (binding) > > > >> > > > >> Cheers, > > > >> > > > >> Chris > > > >> > > > >> On Mon, Jan 26, 2026 at 3:36 PM vaquar khan <[email protected]> > > > >> wrote: > > > >> > > > >> > Hi Josep, > > > >> > > > > >> > Thank you for the detailed response. I appreciate the > clarification > > > >> > regarding the distinction between the Inkless POC and the KIP > > design. > > > >> > > > > >> > However, my objection is not based on temporary bugs in the fork, > > but > > > >> *on > > > >> > architectural gaps in the KIPs themselves* that these > implementation > > > >> issues > > > >> > highlighted. If we are voting to approve the design, the design > > > >> documents > > > >> > must be structurally complete regarding data safety. > > > >> > > > > >> > *1. Regarding Storage Leaks (The Missing Design)* You mentioned > that > > > >> > cleanup logic "can be defined later." However, KIP-1163 explicitly > > > >> > delegates this responsibility to a separate process, and KIP-1165 > > > >> (Object > > > >> > Compaction/GC) is currently marked as "Discarded" in the wiki. > > > >> > > > > >> > We cannot vote to approve a storage engine that has no specified > > > >> mechanism > > > >> > for garbage collection. The "Upload-then-Commit" pattern described > > in > > > >> > KIP-1163 structurally creates orphaned segments during broker > > > failures. > > > >> > Without an active KIP defining the reconciliation protocol (since > > > >> KIP-1165 > > > >> > was withdrawn), the proposal effectively describes a system with > > > >> unbounded > > > >> > storage growth during failure modes. This is a blocking design > gap, > > > not > > > >> an > > > >> > implementation detail. > > > >> > > > > >> > *2. Regarding EOS (The Coordinator Synchronization Gap)* This is > > not a > > > >> > misunderstanding of standard Kafka transactions; it is a critique > of > > > how > > > >> > KIP-1150 changes them. Standard EOS relies on the Partition Leader > > to > > > >> > sequence markers and calculate the LSO (Last Stable Offset) in > > memory. > > > >> > KIP-1150 removes the Leader. > > > >> > > > > >> > KIP-1164 (Batch Coordinator) must explicitly define the RPC flow > > > between > > > >> > the Transaction Coordinator and the Batch Coordinator to replace > the > > > >> > leader's role. Currently, the KIP does not specify how the system > > > >> prevents > > > >> > a "Split Brain" scenario where a consumer reads ahead of a > > transaction > > > >> > marker that hasn't yet been sequenced by the Batch Coordinator. > This > > > is > > > >> a > > > >> > protocol-level correctness issue that must be resolved in the text > > > >> before > > > >> > adoption. > > > >> > > > > >> > Please note - I am maintaining my objection based on missing > > > >> > specifications, not code bugs. > > > >> > > > > >> > I respectfully request that we pause the vote until: > > > >> > > > > >> > A valid design for Garbage Collection (replacing the discarded > > > >> > KIP-1165) is added to the proposal. > > > >> > > > > >> > The Transaction/LSO synchronization protocol is explicitly > > > >> documented > > > >> > in KIP-1164. > > > >> > > > > >> > Regards, > > > >> > > > > >> > Vaquar Khan > > > >> > Sr Data Architect > > > >> > https://www.linkedin.com/in/vaquar-khan-b695577/ > > > >> > > > > >> > > > > > > > > > > -- > > > [image: Aiven] <https://www.aiven.io> > > > > > > *Josep Prat* > > > Sr. Engineering Director, Streaming Services, *Aiven* > > > [email protected] | +491715557497 > > > aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud > > > > > > > <https://www.linkedin.com/company/aiven/> < > > > https://twitter.com/aiven_io> > > > *Aiven Deutschland GmbH* > > > Alexanderufer 3-7, 10117 Berlin > > > > > > Geschäftsführer: Oskari Saarenmaa, Kenneth Chen > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > -- > > Anatolii Popov > > Senior Software Developer, *Aiven OY* > > m: +358505126242 > > w: aiven.io e: [email protected] > > <https://www.facebook.com/aivencloud> > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > > >
