il-kyun opened a new pull request, #23839:
URL: https://github.com/apache/kafka/pull/23839
PartitionStates keeps its order in a LinkedHashMap and moves a partition
to the end with remove + put. HashMap.removeNode does not clear the next
field of the removed entry, so when two partitions of the same bucket
are updated in turn, every removed entry points at the entry created
after it. Once one of them is promoted to the old generation, young GC
treats it as live and copies every removed entry that follows it, and
the chain keeps growing for as long as both partitions receive data.
We hit this on a follower broker (JDK 21, G1, 16 GiB heap). A mixed GC
ran out of free regions (`Survivor regions: 1->0(0)`) and promoted the
fetcher's entries. Young GC pauses then went from 11 ms to 34 ms on
average and stayed there until restart. A heap dump taken 14 hours later
held 2,048,712 removed `LinkedHashMap$Entry` objects alternating between
the two partitions of one bucket of one fetcher's map.
The only condition is two partitions that keep receiving data, fetched
by the same fetcher thread, whose keys fall into the same bucket of that
fetcher's map. The remove + put has been there since KIP-74. Because the
removed entries form a linked list, one GC worker copies them one by one.
Without a broker, updating two such partitions in turn one million times
leaves 2,000,002 entries reachable from the first removed entry.
This change keeps the order in a doubly linked list of nodes looked up
through a HashMap, and moves the existing node to the end instead of
creating a new one. The leader side does the same rotation for fetch
sessions with ImplicitLinkedHashCollection, which also reuses the
element. `LinkedHashMap.putLast` would do this in one call but needs
Java 21, and clients still targets Java 11.
Behavior is unchanged: order, `partitionStateMap()` as a live
unmodifiable view, and `size()`. `stateIterator()` no longer supports
`remove()`. No caller uses it, and with the old code it left `size()`
stale.
Testing:
- New tests in PartitionStatesTest: an update reuses the entry (fails
on the old code), and the map view is ordered, live and unmodifiable.
- PartitionStatesTest, SubscriptionStateTest, AbstractFetcherThreadTest,
ReplicaFetcherThreadTest and ReplicaAlterLogDirsThreadTest pass.
- A randomized comparison against the old class (200,000 mixed
operations, 5 seeds) observed the same order, values and sizes.
New PartitionStatesBenchmark (JDK 21, `-prof gc`, ns/op and B/op):
```
old new
updateAndMoveToEnd 22 11.6 ns, 40 B 8.6 ns, 0 B
updateAndMoveToEnd 125 11.7 ns, 40 B 5.5 ns, 0 B
updateAndMoveToEnd 1000 13.3 ns, 40 B 6.5 ns, 0 B
stateValue 22 2.1 ns 2.3 ns
stateValue 1000 1.7 ns 2.0 ns
forEach 22 17.9 ns 13.2 ns
forEach 1000 1606 ns 1447 ns
```
ReplicaFetcherThreadBenchmark, `testFetcher` (ns/op):
```
partitions old new
100 3,142 +- 490 2,650 +- 34
500 17,843 +- 541 12,900 +- 31
1000 38,029 +- 2,148 32,881 +- 276
5000 229,724 +- 5,314 234,634 +- 11,217
```
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]