Hi Nick, Thanks for the update to the KIP! It's great to see this moving again. Overall the KIP updates LGTM, but I do have a couple of comments
BB1- The KIP specificies using a ConcurrentSkipListMap for the transactional buffer and for IQ READ_UNCOMMITTED queries KS will take a snapshot and during the copy lock the StreamThread with RenetrantReadWriteLock. Now if the IQ queries will only be served from the in-memory transactional buffer, that's an acceptable trade-off since even though the ConcurrentSkipListMap uses weakly consistent iterators, a commit mid-iteration would mean those records not viewed as a result of clearing the buffer. Would it be worth exploring using IQ against the store and the buffer thus removing the copy+locking? I think this could be worthwhile (unless just technically unfeasible) as the way it stands users will have to possibly make tradeoffs in performance by adjusting either the commit interval or the size of uncommitted bytes. Of course as specified in the KIP they have the option of going with READ_COMMITTED. BB2 - In the KIP section "Transaction Buffer Thread Safety" there's this sentence which I find confusing: > Reads from the StreamThread and READ_UNCOMMITTED IQ threads are serviced > by the read buffer. The write buffer is (optionally) used to buffer > writes for commit to the database, if doing so would yield better > performance than using the read buffer. Can you clarify what this means? BB3 - How does the `statestore.uncommitted.max.bytes` work in conjuction with the `statestore.cache.max.bytes.buffering` config? If users elect to go with transactional stores is the statestore cache still needed? Since the statestore cache flushes on commit() as well as the transactional cache what's the workflow like between the two? The default size of the store cache is 10M so if it becomes full and a commit is not called would it empty into the transactional cache? BB4- In discussing the `enable.transactional.statestores` config it mentions "exactly-once, exactly-once-v2 or exactly-once-beta" but we can go with just `exactly-once-v2` , we removed the others since 4.0 Thanks! Bill On Fri, Apr 17, 2026 at 1:13 PM Nick Telford <[email protected]> wrote: > Hi everyone, > > We're circling back to KIP-892, with the goal of including it in 4.4. To > that end, I've updated the KIP with substantial design improvements: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-892:+Transactional+Semantics+for+StateStores > > Highlights: > > - TransactionBuffers that support both RocksDB and InMemory stores > - Support for custom IsolationLevel of Interactive Queries, including a > default (IQv1 and IQv2) and a per-request isolation level (IQv2 only) > - New store-level metric for uncommitted-bytes > > Let me know what you think! > > Regards, > Nick > > On Wed, 17 Apr 2024 at 11:50, Nick Telford <[email protected]> wrote: > > > Hi Walker, > > > > Feel free to ask away, either on the mailing list of the Confluent > > Community Slack, where I hang out :-) > > > > The implementation is *mostly* complete, although it needs some > polishing. > > It's worth noting that KIP-1035 is a hard prerequisite for this. > > > > Regards, > > Nick > > >
