Nicholas Telford created KAFKA-21469:
----------------------------------------

             Summary: NamedCache flush cost and heap usage grow with the 
historical peak of dirty keys
                 Key: KAFKA-21469
                 URL: https://issues.apache.org/jira/browse/KAFKA-21469
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 4.4.0
            Reporter: Nicholas Telford


h2. Problem

{{NamedCache#flush}} and {{NamedCache#clear}} reset the {{dirtyKeys}} 
{{LinkedHashSet}} with {{{}clear(){}}}. {{HashMap#clear()}} nulls every slot of 
its table, and the table never shrinks. Thus each reset costs O(peak dirty 
keys), not O(current dirty keys).

After one flush with many distinct dirty keys (up to the entry limit of the 
cache), every later flush of that cache pays for the large table, for the life 
of the task:
 * *CPU:* after a peak of 100k keys, a flush of 10 keys spends ~100–270 µs on 
{{{}clear(){}}}.
 * *Heap:* the table keeps ~5–11 bytes for each key of the peak. 
{{statestore.cache.max.bytes}} does not count it, and each store cache keeps 
its own peak table, so the total can be larger than the configured budget.

In a production profile of an application with a high cardinality of keys in 
multiple stores, we found a huge fraction of CPU time being spent in 
{{dirtyKeys.clear().}}
h2. Proposed change

Clear in place while the table matches the load. Replace the set when the load 
drops below a quarter of the peak:
{code:java}
private void resetDirtyKeys() {
    final int size = dirtyKeys.size();
    dirtyKeysPeak = Math.max(dirtyKeysPeak, size);
    if (size < dirtyKeysPeak / 4) {
        dirtyKeys = new LinkedHashSet<>();
        dirtyKeysPeak = 0;
    } else {
        dirtyKeys.clear();
    }
}
{code}
Call it from {{flush}} (also in the {{dirtyKeys.isEmpty()}} early return, so an 
idle store releases the table) and from {{{}clear{}}}. The reset cost then 
stays within a fixed multiple of the flush size.

A replacement on every flush ({{{}new LinkedHashSet<>(){}}}) is simpler, but at 
a high steady cardinality it allocates a humongous table on each flush (see 
below).
h2. Benchmark

{{NamedCacheDirtyKeysFlushBenchmark}} (JMH, included with the fix) models one 
put/flush cycle on random distinct keys. {{steady}} varies ±10%; {{burstN}} 
varies ±50% with a 0.1% chance of an N-key cycle. Mean µs for each cycle (JDK 
17, G1, 2 GB heap):
||Workload||Keys||clear() (current)||clear-or-replace||always replace||
|steady|100|20.9|*19.3 (-8%)*|20.8 (-0%)|
|steady|1,000|217|*204 (-6%)*|236 (+9%)|
|steady|100,000|*43,677*|43,764 (+0%)|44,822 (+3%) *|
|burst10k|100|30.8|*24.3 (-21%)*|40.3 (+31%)|
|burst100k|10|288|*72 (-75%)*|88 (-69%)|
|burst100k|1,000|532|*303 (-43%)*|504 (-5%)|

* 17 GCs / 787 ms against 2 GCs / 5–7 ms for the other two strategies, and p99 
+100%.

Across all 11 scenarios, clear-or-replace has the best mean in 9, and it is 
never more than 2% slower than the current code. The benchmark leaves out the 
work of the flush listener, so the gain in production is smaller in the steady 
state.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to