Greetings,

Since KAFKA-16505 (4.1.0), every ProcessorRecordContext carries the source 
record's raw key and value for the exception handlers, and the caching stores 
copy them into each LRUCacheEntry. LRUCacheEntry sizes itself from 
ProcessorRecordContext#residentMemorySizeEstimate(), which does not count them, 
so ThreadCache undercounts by the size of the source record.

So when source values are large and stored values small, the cache never 
reaches statestore.cache.max.bytes and grows with the number of distinct keys. 
We saw this with ~40 KB source records materialized as ~100-byte values: each 
entry was counted at ~160 bytes and held ~40 KB, and after 39 hours the caches 
held 1.9 GB of a 2.1 GB heap until the application ran out of memory.

I am considering that the LRUCacheEntry should consider adding to its size the 
raw key and value lengths.

I can open an PR with some tests cases but here is the short version:

Context: 
https://github.com/apache/kafka/blob/f1e6a31f53a35bee1e7415bcfa2db82391e93159/streams/src/main/java/org/apache/kafka/streams/state/internals/LRUCacheEntry.java#L63-L64

        this.sizeBytes = 1 + // isDirty
            record.residentMemorySizeEstimate() +
            // the source raw record copied into this entry's own context, which
            // ProcessorRecordContext#residentMemorySizeEstimate() does not 
count
            (rawKey == null ? 0 : rawKey.length) +
            (rawValue == null ? 0 : rawValue.length);

Thanks,

Paul

Reply via email to