Stefan Schöffmann created KAFKA-21552:
-----------------------------------------
Summary: Streams record cache retains `sourceRawKey` and
`sourceRawValue` without counting them towards `statestore.cache.max.bytes`
Key: KAFKA-21552
URL: https://issues.apache.org/jira/browse/KAFKA-21552
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Stefan Schöffmann
Attachments: Bildschirmfoto 2026-10-20 um 17.45.41.png,
kafka-streams-memory-issue-main.zip
*Affects versions:* 4.4.0 (reproduced); 4.1.1, 4.2.1, 4.3.0, 4.3.1 (same code
present in the released jars, not reproduced)
*Not affected:* 3.8.1, 3.9.0, 3.9.1, 4.0.0 (no `sourceRawKey` /
`sourceRawValue` yet)
h2. Summary
The state store record cache keeps the raw input record of every cached write
alive, but does not count those bytes towards `statestore.cache.max.bytes`. The
cache can retain many times its configured size and never evict.
h2. Description
Since 4.1, `ProcessorRecordContext` carries the raw bytes of the input record
(`sourceRawKey`, `sourceRawValue`) for exception handlers.
When a processor writes to a store with caching enabled,
`CachingKeyValueStore.putInternal` hands these arrays to the new
`LRUCacheEntry`. The entry's size, which `ThreadCache` uses to decide when to
evict, comes from `ProcessorRecordContext.residentMemorySizeEstimate()`. That
estimate covers timestamp, offset, topic, partition and headers {*}but not the
raw arrays{*}.
So a cached write of a few bytes caused by a large input record is counted as a
few bytes, while the entry keeps the whole input record reachable. Because
eviction only looks at the counted size, a cache that holds large inputs can
stay far below its cap and never evict. The retained heap then no longer
depends on `statestore.cache.max.bytes` but on the size of the input records.
Entries also keep the arrays after a commit flush. They are only marked clean
and stay in the cache as a read cache. As far as I can see, the only place that
clears the raw arrays is `RecordCollectorImpl.send` and it clears the context
passed to the sink, not the one inside the cache entry.
Where to look (4.4.0): `CachingKeyValueStore.putInternal`, `LRUCacheEntry`
constructor, `ProcessorRecordContext.residentMemorySizeEstimate`,
`ThreadCache.maybeEvict`. The window and session caching stores pass the raw
arrays to `LRUCacheEntry` the same way. I only reproduced the key-value store.
h2. Reproduction
The attached Maven project needs JDK 17+, Maven and no broker
(`TopologyTestDriver`):
```
mvn -q compile exec:exec
```
It builds a Processor API topology `source -> processor -> sink` with one store
that has `.withCachingEnabled()`. Each of 400 input records is 1 MiB with a
unique key. The processor stores a 4-byte value and forwards a 1-byte record,
so the only large objects are the raw inputs. The cache cap is 32 MiB. After
the run it reads the cache's own counter and its entries by reflection.
Output with kafka-streams 4.4.0:
```
statestore.cache.max.bytes : 33,554,432 B
input records : 400 x 1,048,576 B = 400 MiB
cache counts (sizeInBytes) : 23,090 B
cache entries : 400 (400 reference a sourceRawValue, 400 MiB)
heap in use after GC : 409 MiB
```
What this shows:
- The cache counts 23 KB against a 32 MiB cap, so it never evicts.
- All 400 entries reference their 1 MiB input record and the heap holds all 400
MiB.
- With the cache disabled (`mvn -q exec:exec -Dcap=0`) every entry is evicted
on `put`: 0 entries and 7 MiB heap. The retained memory is therefore caused by
the cache.
The ratio is exaggerated on purpose (about 58 counted bytes against 1 MiB per
entry).
The `heap in use after GC` line depends on the collector (the pom pins
`-XX:+UseSerialGC`). The `cache counts` and `cache entries` lines do not.
h2. Seen in production
We first saw this in a production application on 4.3.x (Processor API, many
cached stores, `statestore.cache.max.bytes` = 200 MiB). Old-gen usage kept
growing until G1 ran back-to-back full GCs. In heap dumps the `ThreadCache`
retained up to 85% of the heap. In the two dumps where we read the counter, the
cache held about 13.5x what it reported, which is why no eviction ever started.
Heap dump screenshots are attached.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)