Hi Vaquar, The Exactly Once design is still underspecified, and unimplemented. Your checks against the 4.4.0-SNAPSHOT codebase are invalid, and nonsensical. Please discontinue using LLMs to review KIPs. Myself and other members of the Kafka mailing list expect human reviewers to bring relevant comments to KIP discussions; If we were interested in reading LLM reviews we would consult them ourselves.
Thanks, Greg On Sun, Jul 12, 2026 at 7:54 PM vaquar khan <[email protected]> wrote: > Hi Ivan, Giuseppe, Greg, and all, > > I am following up on the commitment I made in the KIP-1150 VOTE thread. > A little history so it is easy to follow: > > - 2026-01-26 (KIP-1150 vote): I objected on two design gaps - > missing storage GC, and the exactly-once (EOS) coordinator-sync gap - and > asked to pause. > - 2026-02-25 (KIP-1150 vote): I withdrew the storage-leak objection > after the KIP-1163 reconciliation loop was added, then withdraw my > objection entirely and voted +1 (non-binding), on the understanding > that the EOS details would be finalized in KIP-1164. > > > *I am now moving those EOS questions here, as promised.* > > *RESOLVED (acknowledging on the record)* - *Orphan files / storage leak: > *the > WAL file management and Orphan files sections, the owner-handover protocol, > and DisklessDescribeFiles (HasLiveBatches) address this. Thank you. > *- Idempotent produce*: DisklessCommitFile carrying > ProducerId/ProducerEpoch/BaseSequence/LastSequence with an IsDuplicate > result is exactly right. > > > *STILL OPEN - Exactly-once (LSO + transaction protocol)*KIP-1164 states > that "Some operations remain out of scope of this KIP, such as the ones for > transaction management," while DisklessCommitFile says it performs > "transactional checks." Those two lines are in slight tension, and that gap > is where read_committed correctness lives. One small thing to align first: > the DC role text says it manages "producer and transaction state," which > reads against the "out of scope" note - a sentence on where the boundary > sits would help. > > > *Concretely, checked against 4.4.0-SNAPSHOT:* > * 1. No LSO in the read path*. DisklessFindBatchesResponse > returns LogStartOffset and HighWatermark, but no LastStableOffset. In > classic Kafka a read_committed consumer only reads up to the LSO, and > the LSO is computed on the partition leader from in-memory producer state: > > ProducerStateManager.firstUnstableOffset() derived from the > ongoingTxns and unreplicatedTxns maps > > (storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManager.java). > > KIP-1150 removes the leader, so where is ongoing-transaction state > tracked, how does the Diskless Coordinator compute the LSO, and how is it > returned to the fetching broker? > > 2. No aborted-transaction data. A read_committed Fetch response must > carry the list of aborted transactions so the consumer can skip > them (FetchResponse AbortedTransactions, v4+). > DisklessFindBatches returns batch coordinates but no aborted-transaction > metadata. Where does that come from in the diskless path? > > 3. No Transaction-Coordinator to Diskless-Coordinator > protocol.DisklessCommitFile carries ProducerId/ProducerEpoch for > idempotence, but there are no transaction control-marker records > (commit/abort) and no defined handshake with the Transaction Coordinator. > When the Transaction Coordinator writes a commit/abort marker, what flow > lets the DC confirm it has sequenced all data batches for that > producer/epoch before the marker takes effect? > > 4. The delayed-data race. DisklessCommitFile assigns offsets at > commit time, and a producer's upload and commit are separate steps. If a > broker stalls after uploading a transaction's data batch but before > committing its coordinates, and the commit marker is sequenced at the DC > first, does the DC hold until the data is committed, or can it advance past > a transaction whose data is not yet visible? That would be a read_committed > isolation violation if unhandled. > > I am not asking to design all of the EOS in this KIP. But "out of scope" > plus "transactional checks" leaves the correctness-critical piece undefined > across every diskless KIP. Could we either (a) specify the LSO computation, > the aborted-transaction path, and the Txn-Coordinator/DC protocol here, > or (b) name the specific KIP that will own them and mark > DisklessCommitFile's "transactional checks" as provisional until that KIP > lands? I would rather we not vote on the coordinator contract with that > piece pointing at a document that does not exist yet. > > Happy to help review whichever KIP takes this on - it is also close to > the KIP-1279 transaction-marker work I have been in. > > Regards, > Vaquar Khan > LinkedIn - https://www.linkedin.com/in/vaquar-khan-b695577/ >
