Nicholas Telford created KAFKA-21469:
----------------------------------------
Summary: NamedCache flush cost and heap usage grow with the
historical peak of dirty keys
Key: KAFKA-21469
URL: https://issues.apache.org/jira/browse/KAFKA-21469
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Nicholas Telford
h2. Problem
{{NamedCache#flush}} and {{NamedCache#clear}} reset the {{dirtyKeys}}
{{LinkedHashSet}} with {{{}clear(){}}}. {{HashMap#clear()}} nulls every slot of
its table, and the table never shrinks. Thus each reset costs O(peak dirty
keys), not O(current dirty keys).
After one flush with many distinct dirty keys (up to the entry limit of the
cache), every later flush of that cache pays for the large table, for the life
of the task:
* *CPU:* after a peak of 100k keys, a flush of 10 keys spends ~100–270 µs on
{{{}clear(){}}}.
* *Heap:* the table keeps ~5–11 bytes for each key of the peak.
{{statestore.cache.max.bytes}} does not count it, and each store cache keeps
its own peak table, so the total can be larger than the configured budget.
In a production profile of an application with a high cardinality of keys in
multiple stores, we found a huge fraction of CPU time being spent in
{{dirtyKeys.clear().}}
h2. Proposed change
Clear in place while the table matches the load. Replace the set when the load
drops below a quarter of the peak:
{code:java}
private void resetDirtyKeys() {
final int size = dirtyKeys.size();
dirtyKeysPeak = Math.max(dirtyKeysPeak, size);
if (size < dirtyKeysPeak / 4) {
dirtyKeys = new LinkedHashSet<>();
dirtyKeysPeak = 0;
} else {
dirtyKeys.clear();
}
}
{code}
Call it from {{flush}} (also in the {{dirtyKeys.isEmpty()}} early return, so an
idle store releases the table) and from {{{}clear{}}}. The reset cost then
stays within a fixed multiple of the flush size.
A replacement on every flush ({{{}new LinkedHashSet<>(){}}}) is simpler, but at
a high steady cardinality it allocates a humongous table on each flush (see
below).
h2. Benchmark
{{NamedCacheDirtyKeysFlushBenchmark}} (JMH, included with the fix) models one
put/flush cycle on random distinct keys. {{steady}} varies ±10%; {{burstN}}
varies ±50% with a 0.1% chance of an N-key cycle. Mean µs for each cycle (JDK
17, G1, 2 GB heap):
||Workload||Keys||clear() (current)||clear-or-replace||always replace||
|steady|100|20.9|*19.3 (-8%)*|20.8 (-0%)|
|steady|1,000|217|*204 (-6%)*|236 (+9%)|
|steady|100,000|*43,677*|43,764 (+0%)|44,822 (+3%) *|
|burst10k|100|30.8|*24.3 (-21%)*|40.3 (+31%)|
|burst100k|10|288|*72 (-75%)*|88 (-69%)|
|burst100k|1,000|532|*303 (-43%)*|504 (-5%)|
* 17 GCs / 787 ms against 2 GCs / 5–7 ms for the other two strategies, and p99
+100%.
Across all 11 scenarios, clear-or-replace has the best mean in 9, and it is
never more than 2% slower than the current code. The benchmark leaves out the
work of the flush listener, so the gain in production is smaller in the steady
state.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)