[
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126989#comment-18126989
]
Eric Chang commented on KAFKA-19992:
------------------------------------
Hi [~stoeckmk], one more note: while looking into this, I found KAFKA-21276
(PR: https://github.com/apache/kafka/pull/23815).
If your tables are read with {{builder.table(topic)}},
{{topology.optimization}} is {{all}} or {{reuse.ktable.source.topics}}, and you
run standby replicas, then a standby task keeps fetching the input topic beyond
the offset committed by the active task and keeps those records in memory.
During a backfill this can lead to an OutOfMemoryError like the one you
described.
If your setup is different, the questions above are still very helpful to find
the cause.
> Allow to configure max memory for restore path
> ----------------------------------------------
>
> Key: KAFKA-19992
> URL: https://issues.apache.org/jira/browse/KAFKA-19992
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Muralidhar Basani
> Priority: Major
> Labels: kip
> Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The
> `StoreChangelogReader` fetches a batch of message, and buffers them inside
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would
> require a KIP though). – Even if we re-use the existing config, not 100% sure
> if we would need a KIP or not?
> KIP-1325:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)