Hi Greg, One clarification first, since it matters to me: these were my own comments. The lines I quote are from KIP-1164; where I mention trunk, it was only the current read_committed behaviour I used as the baseline to compare the proposed diskless contract against.
I think that is where the disconnect came from: my earlier note was misread as a claim that diskless transactions already exist in the trunk.That was not the claim, so the checks were not wrong, they were just misread. We are aligned on the substance. The transaction semantics are deferred to a later KIP, and that was exactly my point. To be precise about what I was flagging: the role section describes the coordinator as managing "producer and transaction state for Diskless topics," and DisklessCommitFile performs checks "for producer idempotence, transactional checks," while the KIP also states that "Some operations remain out of scope of this KIP, such as the ones for transaction management." I was pointing at that seam. The contract already names transactional checks, but the semantics are left to a future document. The idempotence path reads correctly to me: DisklessCommitFile carries ProducerId, ProducerEpoch, BaseSequence and LastSequence with an IsDuplicate result. The read_committed path is the open piece. DisklessFindBatchesResponse returns LogStartOffset and HighWatermark but no last stable offset, and no aborted-transaction list. My reference to trunk was only to point at what a read_committed consumer relies on today, the LSO and the aborted-transaction list a FetchResponse carries, not a claim that the diskless side has it. Since that piece is deferred, I would like to help when it is picked up, whether that is the LSO and aborted-transaction handling as a team or just reviewing what you and the others draft. Regards, Vaquar Khan https://www.linkedin.com/in/vaquar-khan-b695577/ On Mon, 13 Jul 2026 at 18:38, Greg Harris via dev <[email protected]> wrote: > Hi Vaquar, > > The Exactly Once design is still underspecified, and unimplemented. Your > checks against the 4.4.0-SNAPSHOT codebase are invalid, and nonsensical. > Please discontinue using LLMs to review KIPs. Myself and other members of > the Kafka mailing list expect human reviewers to bring relevant comments to > KIP discussions; If we were interested in reading LLM reviews we would > consult them ourselves. > > Thanks, > Greg > > On Sun, Jul 12, 2026 at 7:54 PM vaquar khan <[email protected]> wrote: > > > Hi Ivan, Giuseppe, Greg, and all, > > > > I am following up on the commitment I made in the KIP-1150 VOTE thread. > > A little history so it is easy to follow: > > > > - 2026-01-26 (KIP-1150 vote): I objected on two design gaps - > > missing storage GC, and the exactly-once (EOS) coordinator-sync gap - and > > asked to pause. > > - 2026-02-25 (KIP-1150 vote): I withdrew the storage-leak objection > > after the KIP-1163 reconciliation loop was added, then withdraw my > > objection entirely and voted +1 (non-binding), on the understanding > > that the EOS details would be finalized in KIP-1164. > > > > > > *I am now moving those EOS questions here, as promised.* > > > > *RESOLVED (acknowledging on the record)* - *Orphan files / storage leak: > > *the > > WAL file management and Orphan files sections, the owner-handover > protocol, > > and DisklessDescribeFiles (HasLiveBatches) address this. Thank you. > > *- Idempotent produce*: DisklessCommitFile carrying > > ProducerId/ProducerEpoch/BaseSequence/LastSequence with an IsDuplicate > > result is exactly right. > > > > > > *STILL OPEN - Exactly-once (LSO + transaction protocol)*KIP-1164 states > > that "Some operations remain out of scope of this KIP, such as the ones > for > > transaction management," while DisklessCommitFile says it performs > > "transactional checks." Those two lines are in slight tension, and that > gap > > is where read_committed correctness lives. One small thing to align > first: > > the DC role text says it manages "producer and transaction state," which > > reads against the "out of scope" note - a sentence on where the boundary > > sits would help. > > > > > > *Concretely, checked against 4.4.0-SNAPSHOT:* > > * 1. No LSO in the read path*. DisklessFindBatchesResponse > > returns LogStartOffset and HighWatermark, but no LastStableOffset. In > > classic Kafka a read_committed consumer only reads up to the LSO, and > > the LSO is computed on the partition leader from in-memory producer > state: > > > > ProducerStateManager.firstUnstableOffset() derived from the > > ongoingTxns and unreplicatedTxns maps > > > > > (storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManager.java). > > > > KIP-1150 removes the leader, so where is ongoing-transaction state > > tracked, how does the Diskless Coordinator compute the LSO, and how is > it > > returned to the fetching broker? > > > > 2. No aborted-transaction data. A read_committed Fetch response must > > carry the list of aborted transactions so the consumer can skip > > them (FetchResponse AbortedTransactions, v4+). > > DisklessFindBatches returns batch coordinates but no aborted-transaction > > metadata. Where does that come from in the diskless path? > > > > 3. No Transaction-Coordinator to Diskless-Coordinator > > protocol.DisklessCommitFile carries ProducerId/ProducerEpoch for > > idempotence, but there are no transaction control-marker records > > (commit/abort) and no defined handshake with the Transaction Coordinator. > > When the Transaction Coordinator writes a commit/abort marker, what flow > > lets the DC confirm it has sequenced all data batches for that > > producer/epoch before the marker takes effect? > > > > 4. The delayed-data race. DisklessCommitFile assigns offsets at > > commit time, and a producer's upload and commit are separate steps. If a > > broker stalls after uploading a transaction's data batch but before > > committing its coordinates, and the commit marker is sequenced at the DC > > first, does the DC hold until the data is committed, or can it advance > past > > a transaction whose data is not yet visible? That would be a > read_committed > > isolation violation if unhandled. > > > > I am not asking to design all of the EOS in this KIP. But "out of scope" > > plus "transactional checks" leaves the correctness-critical piece > undefined > > across every diskless KIP. Could we either (a) specify the LSO > computation, > > the aborted-transaction path, and the Txn-Coordinator/DC protocol here, > > or (b) name the specific KIP that will own them and mark > > DisklessCommitFile's "transactional checks" as provisional until that KIP > > lands? I would rather we not vote on the coordinator contract with that > > piece pointing at a document that does not exist yet. > > > > Happy to help review whichever KIP takes this on - it is also close to > > the KIP-1279 transaction-marker work I have been in. > > > > Regards, > > Vaquar Khan > > LinkedIn - https://www.linkedin.com/in/vaquar-khan-b695577/ > > >
