Greetings, Since KAFKA-16505 (4.1.0), every ProcessorRecordContext carries the source record's raw key and value for the exception handlers, and the caching stores copy them into each LRUCacheEntry. LRUCacheEntry sizes itself from ProcessorRecordContext#residentMemorySizeEstimate(), which does not count them, so ThreadCache undercounts by the size of the source record.
So when source values are large and stored values small, the cache never reaches statestore.cache.max.bytes and grows with the number of distinct keys. We saw this with ~40 KB source records materialized as ~100-byte values: each entry was counted at ~160 bytes and held ~40 KB, and after 39 hours the caches held 1.9 GB of a 2.1 GB heap until the application ran out of memory. I am considering that the LRUCacheEntry should consider adding to its size the raw key and value lengths. I can open an PR with some tests cases but here is the short version: Context: https://github.com/apache/kafka/blob/f1e6a31f53a35bee1e7415bcfa2db82391e93159/streams/src/main/java/org/apache/kafka/streams/state/internals/LRUCacheEntry.java#L63-L64 this.sizeBytes = 1 + // isDirty record.residentMemorySizeEstimate() + // the source raw record copied into this entry's own context, which // ProcessorRecordContext#residentMemorySizeEstimate() does not count (rawKey == null ? 0 : rawKey.length) + (rawValue == null ? 0 : rawValue.length); Thanks, Paul
