dao-jun opened a new pull request, #23997: URL: https://github.com/apache/kafka/pull/23997
### Motivation `TransactionIndex.collectAbortedTxns` sequentially reads and deserializes every index record from the beginning of the file. For a `READ_COMMITTED` fetch near the end of a segment with many aborted transactions, this can decode tens of thousands of records whose `lastOffset` is too old to overlap the fetch. ### Changes - Use binary search over the fixed-size records to find the first potentially overlapping record by `lastOffset` and the original termination point by `lastStableOffset`. - Retain a sequential path for fetches that overlap the first record, reusing the decoded first record to avoid regressions for reads at the beginning of the index. - Share the positional record decoder with the existing iterator. The on-disk format, public API, and per-record heap usage are unchanged. Both search keys are non-decreasing for valid Kafka-generated indexes: abort markers are appended in log-offset order, while LSO is the earliest remaining transaction's first offset, or the completed marker's offset plus one. A new out-of-order transaction-completion test covers this ordering precondition. The new tests also compare records, order, and `isComplete` against the sequential implementation over 1,000 generated fetch ranges, including the case where the LSO termination point precedes the first overlapping transaction. ### Tests - `./gradlew :storage:test --tests org.apache.kafka.storage.internals.log.TransactionIndexTest --tests org.apache.kafka.storage.internals.log.ProducerStateManagerTest.testLastStableOffsetDoesNotRegressWhenTransactionsCompleteOutOfOrder --tests 'org.apache.kafka.storage.internals.log.UnifiedLogTest.*TransactionIndex*' --tests 'org.apache.kafka.storage.internals.log.UnifiedLogTest.*ReadCommitted*' --tests 'org.apache.kafka.storage.internals.log.UnifiedLogTest.*Aborted*'` - `./gradlew spotlessApply checkstyleMain checkstyleTest spotlessCheck` ### Benchmark The included JMH benchmark was run unchanged before and after the optimization on Java 25, with 2 forks, 2 warmups, and 3 measurement iterations of 1 second each. | Index size | Fetch position | Scan size | Baseline ns/op | Optimized ns/op | Speedup | | ---: | ---: | ---: | ---: | ---: | ---: | | 16 | 0% | 1 | 1,145.1 | 891.9 | 1.28x | | 1,024 | 99% | 1 | 544,086.0 | 7,525.1 | 72.30x | | 65,536 | 0% | 1 | 1,120.1 | 893.4 | 1.25x | | 65,536 | 50% | 1 | 17,567,046.0 | 11,679.0 | 1,504.16x | | 65,536 | 99% | 1 | 35,172,380.1 | 11,618.2 | 3,027.36x | All 18 parameter combinations improved, with non-overlapping 99.9% confidence intervals. These are hot-page-cache index-lookup microbenchmarks, not end-to-end fetch measurements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
