[
https://issues.apache.org/jira/browse/KAFKA-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-21469:
---------------------------------------
Assignee: Nicholas Telford
> NamedCache flush cost and heap usage grow with the historical peak of dirty
> keys
> --------------------------------------------------------------------------------
>
> Key: KAFKA-21469
> URL: https://issues.apache.org/jira/browse/KAFKA-21469
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.4.0
> Reporter: Nicholas Telford
> Assignee: Nicholas Telford
> Priority: Minor
>
> h2. Problem
> {{NamedCache#flush}} and {{NamedCache#clear}} reset the {{dirtyKeys}}
> {{LinkedHashSet}} with {{{}clear(){}}}. {{HashMap#clear()}} nulls every slot
> of its table, and the table never shrinks. Thus each reset costs O(peak dirty
> keys), not O(current dirty keys).
> After one flush with many distinct dirty keys (up to the entry limit of the
> cache), every later flush of that cache pays for the large table, for the
> life of the task:
> * *CPU:* after a peak of 100k keys, a flush of 10 keys spends ~100–270 µs on
> {{{}clear(){}}}.
> * *Heap:* the table keeps ~5–11 bytes for each key of the peak.
> {{statestore.cache.max.bytes}} does not count it, and each store cache keeps
> its own peak table, so the total can be larger than the configured budget.
> In a production profile of an application with a high cardinality of keys in
> multiple stores, we found a huge fraction of CPU time being spent in
> {{dirtyKeys.clear().}}
> h2. Proposed change
> Clear in place while the table matches the load. Replace the set when the
> load drops below a quarter of the peak:
> {code:java}
> private void resetDirtyKeys() {
> final int size = dirtyKeys.size();
> dirtyKeysPeak = Math.max(dirtyKeysPeak, size);
> if (size < dirtyKeysPeak / 4) {
> dirtyKeys = new LinkedHashSet<>();
> dirtyKeysPeak = 0;
> } else {
> dirtyKeys.clear();
> }
> }
> {code}
> Call it from {{flush}} (also in the {{dirtyKeys.isEmpty()}} early return, so
> an idle store releases the table) and from {{{}clear{}}}. The reset cost then
> stays within a fixed multiple of the flush size.
> A replacement on every flush ({{{}new LinkedHashSet<>(){}}}) is simpler, but
> at a high steady cardinality it allocates a humongous table on each flush
> (see below).
> h2. Benchmark
> {{NamedCacheDirtyKeysFlushBenchmark}} (JMH, included with the fix) models one
> put/flush cycle on random distinct keys. {{steady}} varies ±10%; {{burstN}}
> varies ±50% with a 0.1% chance of an N-key cycle. Mean µs for each cycle (JDK
> 17, G1, 2 GB heap):
> ||Workload||Keys||clear() (current)||clear-or-replace||always replace||
> |steady|100|20.9|*19.3 (-8%)*|20.8 (-0%)|
> |steady|1,000|217|*204 (-6%)*|236 (+9%)|
> |steady|100,000|*43,677*|43,764 (+0%)|44,822 (+3%) *|
> |burst10k|100|30.8|*24.3 (-21%)*|40.3 (+31%)|
> |burst100k|10|288|*72 (-75%)*|88 (-69%)|
> |burst100k|1,000|532|*303 (-43%)*|504 (-5%)|
> * 17 GCs / 787 ms against 2 GCs / 5–7 ms for the other two strategies, and
> p99 +100%.
> Across all 11 scenarios, clear-or-replace has the best mean in 9, and it is
> never more than 2% slower than the current code. The benchmark leaves out the
> work of the flush listener, so the gain in production is smaller in the
> steady state.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)