[ 
https://issues.apache.org/jira/browse/KAFKA-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-21469:
---------------------------------------

    Assignee: Nicholas Telford

> NamedCache flush cost and heap usage grow with the historical peak of dirty 
> keys
> --------------------------------------------------------------------------------
>
>                 Key: KAFKA-21469
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21469
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.4.0
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>
> h2. Problem
> {{NamedCache#flush}} and {{NamedCache#clear}} reset the {{dirtyKeys}} 
> {{LinkedHashSet}} with {{{}clear(){}}}. {{HashMap#clear()}} nulls every slot 
> of its table, and the table never shrinks. Thus each reset costs O(peak dirty 
> keys), not O(current dirty keys).
> After one flush with many distinct dirty keys (up to the entry limit of the 
> cache), every later flush of that cache pays for the large table, for the 
> life of the task:
>  * *CPU:* after a peak of 100k keys, a flush of 10 keys spends ~100–270 µs on 
> {{{}clear(){}}}.
>  * *Heap:* the table keeps ~5–11 bytes for each key of the peak. 
> {{statestore.cache.max.bytes}} does not count it, and each store cache keeps 
> its own peak table, so the total can be larger than the configured budget.
> In a production profile of an application with a high cardinality of keys in 
> multiple stores, we found a huge fraction of CPU time being spent in 
> {{dirtyKeys.clear().}}
> h2. Proposed change
> Clear in place while the table matches the load. Replace the set when the 
> load drops below a quarter of the peak:
> {code:java}
> private void resetDirtyKeys() {
>     final int size = dirtyKeys.size();
>     dirtyKeysPeak = Math.max(dirtyKeysPeak, size);
>     if (size < dirtyKeysPeak / 4) {
>         dirtyKeys = new LinkedHashSet<>();
>         dirtyKeysPeak = 0;
>     } else {
>         dirtyKeys.clear();
>     }
> }
> {code}
> Call it from {{flush}} (also in the {{dirtyKeys.isEmpty()}} early return, so 
> an idle store releases the table) and from {{{}clear{}}}. The reset cost then 
> stays within a fixed multiple of the flush size.
> A replacement on every flush ({{{}new LinkedHashSet<>(){}}}) is simpler, but 
> at a high steady cardinality it allocates a humongous table on each flush 
> (see below).
> h2. Benchmark
> {{NamedCacheDirtyKeysFlushBenchmark}} (JMH, included with the fix) models one 
> put/flush cycle on random distinct keys. {{steady}} varies ±10%; {{burstN}} 
> varies ±50% with a 0.1% chance of an N-key cycle. Mean µs for each cycle (JDK 
> 17, G1, 2 GB heap):
> ||Workload||Keys||clear() (current)||clear-or-replace||always replace||
> |steady|100|20.9|*19.3 (-8%)*|20.8 (-0%)|
> |steady|1,000|217|*204 (-6%)*|236 (+9%)|
> |steady|100,000|*43,677*|43,764 (+0%)|44,822 (+3%) *|
> |burst10k|100|30.8|*24.3 (-21%)*|40.3 (+31%)|
> |burst100k|10|288|*72 (-75%)*|88 (-69%)|
> |burst100k|1,000|532|*303 (-43%)*|504 (-5%)|
> * 17 GCs / 787 ms against 2 GCs / 5–7 ms for the other two strategies, and 
> p99 +100%.
> Across all 11 scenarios, clear-or-replace has the best mean in 9, and it is 
> never more than 2% slower than the current code. The benchmark leaves out the 
> work of the flush listener, so the gain in production is smaller in the 
> steady state.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to