ILKYUN LIM created KAFKA-21316:
----------------------------------
Summary: Removed entries in PartitionStates stay chained and slow
down young GC on follower brokers
Key: KAFKA-21316
URL: https://issues.apache.org/jira/browse/KAFKA-21316
Project: Kafka
Issue Type: Bug
Components: clients, replication
Affects Versions: 4.3.1
Reporter: ILKYUN LIM
Assignee: ILKYUN LIM
h3. Problem
{{PartitionStates}} keeps its order in a {{LinkedHashMap}} and moves a
partition to the end with {{remove}} + {{put}} ({{updateAndMoveToEnd}}).
{{HashMap.removeNode}} does not clear the {{next}} field of the removed entry
(same in JDK 17, 21 and 25). When two partitions whose keys fall into the same
bucket are updated in turn, every removed entry keeps pointing at the entry
created after it, so the removed entries form one chain.
Once one entry of that chain is promoted to the old generation, every young GC
treats the chain as live and copies all removed entries that follow it. The
chain keeps growing as long as both partitions receive data. Because it is a
linked list, a single GC worker copies it entry by entry, so pauses grow while
the copied bytes stay small.
The only condition is two partitions that keep receiving data, fetched by the
same fetcher thread, whose keys fall into the same bucket of that fetcher's
map. The remove + put pattern has been there since KIP-74.
h3. Observed
Follower broker, Kafka 4.3.1, JDK 21, G1, 16 GiB heap, {{MaxGCPauseMillis=20}}:
* A mixed GC ran out of free regions ({{Survivor regions: 1->0(0)}}) and
promoted the replica fetcher's map entries.
* Young GC pauses went from 11 ms to 34 ms on average (single pauses 30-50 ms)
and stayed there until the broker was restarted. G1 then shrank Eden to keep up
with the pause goal, which made young GCs about 6 times more frequent.
* A heap dump taken about 14 hours after the onset held 2,048,712 removed
{{LinkedHashMap$Entry}} objects linked through {{next}}, alternating between
the two partitions of one bucket of one fetcher's map.
Reproduction without a broker: updating two such partitions in turn one million
times leaves 2,000,002 entries reachable from the first removed entry.
h3. Fix
Pull request: https://github.com/apache/kafka/pull/23839
Keep the order in a doubly linked list of nodes looked up through a
{{HashMap}}, and move the existing node to the end instead of creating a new
one. This matches what the leader side already does for fetch sessions with
{{ImplicitLinkedHashCollection}}. {{LinkedHashMap.putLast}} would do the same
in one call but needs Java 21, and the clients module still targets Java 11.
Order, {{partitionStateMap()}} (a live unmodifiable view) and {{size()}} behave
as before. {{stateIterator().remove()}} is no longer supported; no caller uses
it, and with the old code it left {{size()}} stale.
A new JMH benchmark ({{PartitionStatesBenchmark}}, JDK 21) shows
{{updateAndMoveToEnd}} going from 11.6-13.3 ns and 40 B/op to 5.5-8.6 ns and 0
B/op. {{ReplicaFetcherThreadBenchmark}} is faster at 100-1000 partitions and
within the error at 5000.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
