nicktelford opened a new pull request, #23999: URL: https://github.com/apache/kafka/pull/23999
`RocksDBTransactionBuffer` writes every staged write twice: to `pendingWrites` on the heap, which serves reads, and to a native RocksDB `WriteBatch`. The `WriteBatch` holds native memory for the whole transaction, and the buffer changes it while it holds the write lock. The buffer now stages writes on the heap only. Writes to other column families and range deletes also stay on the heap until the commit. At commit, `prepareFlush()` builds the `WriteBatch` from the staged state outside the write lock, and `flushToBase()` writes it to RocksDB under the write lock. A JMH model of the two designs measured writes 2 to 7 times faster, and commits 8 to 12 times faster when the transaction overwrites keys. A commit with no overwrites and 1000-byte values is up to 28% slower. Because no buffer writes to a backend at stage time now, the `stageToBackend` hook of `AbstractTransactionBuffer` has no function. This change removes it and all of its implementations. The uncommitted-bytes estimate also counted a key once for each write in a transaction, so a key written many times could cause an early commit with little staged data. `AbstractTransactionBuffer.stage` now subtracts the size of the previous entry for the key before it adds the new entry. `RocksDBTransactionBuffer` uses this shared accounting for all of its staged data. `statestore.uncommitted.max.bytes` now counts each staged write once. Before, it counted the heap copy and the `WriteBatch` copy. The commit still needs a native `WriteBatch`, but only for one store at a time, because stores commit one after the other. The extra memory at commit is therefore at most the uncommitted bytes of the largest store. A thread now commits early when its uncommitted bytes plus the uncommitted bytes of its largest store exceed its share of the limit. The global thread uses the same check. The limit is thus again a bound on the memory at commit. The check uses the largest value of `StateStore.approximateNumUncommittedBytes()`, so it needs no change to a public interface. In-memory stores need no extra memory at commit, so applications with only in-memory stores commit early after less data than in 4.4. Applications with more than one transactional RocksDB store can buffer more data before an early commit. The 4.5.0 upgrade guide and the documentation of the config record this. The default stays at 64 MB. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
