il-kyun opened a new pull request, #23839:
URL: https://github.com/apache/kafka/pull/23839

   PartitionStates keeps its order in a LinkedHashMap and moves a partition
   to the end with remove + put. HashMap.removeNode does not clear the next
   field of the removed entry, so when two partitions of the same bucket
   are updated in turn, every removed entry points at the entry created
   after it. Once one of them is promoted to the old generation, young GC
   treats it as live and copies every removed entry that follows it, and
   the chain keeps growing for as long as both partitions receive data.
   
   We hit this on a follower broker (JDK 21, G1, 16 GiB heap). A mixed GC
   ran out of free regions (`Survivor regions: 1->0(0)`) and promoted the
   fetcher's entries. Young GC pauses then went from 11 ms to 34 ms on
   average and stayed there until restart. A heap dump taken 14 hours later
   held 2,048,712 removed `LinkedHashMap$Entry` objects alternating between
   the two partitions of one bucket of one fetcher's map.
   
   The only condition is two partitions that keep receiving data, fetched
   by the same fetcher thread, whose keys fall into the same bucket of that
   fetcher's map. The remove + put has been there since KIP-74. Because the
   removed entries form a linked list, one GC worker copies them one by one.
   Without a broker, updating two such partitions in turn one million times
   leaves 2,000,002 entries reachable from the first removed entry.
   
   This change keeps the order in a doubly linked list of nodes looked up
   through a HashMap, and moves the existing node to the end instead of
   creating a new one. The leader side does the same rotation for fetch
   sessions with ImplicitLinkedHashCollection, which also reuses the
   element. `LinkedHashMap.putLast` would do this in one call but needs
   Java 21, and clients still targets Java 11.
   
   Behavior is unchanged: order, `partitionStateMap()` as a live
   unmodifiable view, and `size()`. `stateIterator()` no longer supports
   `remove()`. No caller uses it, and with the old code it left `size()`
   stale.
   
   Testing:
   - New tests in PartitionStatesTest: an update reuses the entry (fails
     on the old code), and the map view is ordered, live and unmodifiable.
   - PartitionStatesTest, SubscriptionStateTest, AbstractFetcherThreadTest,
     ReplicaFetcherThreadTest and ReplicaAlterLogDirsThreadTest pass.
   - A randomized comparison against the old class (200,000 mixed
     operations, 5 seeds) observed the same order, values and sizes.
   
   New PartitionStatesBenchmark (JDK 21, `-prof gc`, ns/op and B/op):
   
   ```
                           old            new
   updateAndMoveToEnd 22   11.6 ns, 40 B   8.6 ns, 0 B
   updateAndMoveToEnd 125  11.7 ns, 40 B   5.5 ns, 0 B
   updateAndMoveToEnd 1000 13.3 ns, 40 B   6.5 ns, 0 B
   stateValue 22            2.1 ns         2.3 ns
   stateValue 1000          1.7 ns         2.0 ns
   forEach 22              17.9 ns        13.2 ns
   forEach 1000            1606 ns        1447 ns
   ```
   
   ReplicaFetcherThreadBenchmark, `testFetcher` (ns/op):
   
   ```
   partitions  old              new
   100         3,142 +- 490     2,650 +- 34
   500         17,843 +- 541    12,900 +- 31
   1000        38,029 +- 2,148  32,881 +- 276
   5000        229,724 +- 5,314 234,634 +- 11,217
   ```
   
   Co-Authored-By: Claude Opus 5.5 <[email protected]>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to