[
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-19992:
------------------------------------
Description:
Kafka Streams allows users to configure many different memory limits, to
pro-actively manage memory usage to avoid OutOfMemory errors.
However, on the restore code path, there is no such memory management. The
`StoreChangelogReader` fetches a batch of message, and buffers them inside
`ChangelogMetadata`, and there is no limit enforced on this buffer.
We could either re-use the existing config for input topic buffer (`
buffered.records.per.partition`) or introduce a new config (this would require
a KIP though). – Even if we re-use the existing config, not 100% sure if we
would need a KIP or not?
KIP-1325:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
was:
Kafka Streams allows users to configure many different memory limits, to
pro-actively manage memory usage to avoid OutOfMemory errors.
However, on the restore code path, there is no such memory management. The
`StoreChangelogReader` fetches a batch of message, and buffers them inside
`ChangelogMetadata`, and there is no limit enforced on this buffer.
We could either re-use the existing config for input topic buffer (`
buffered.records.per.partition`) or introduce a new config (this would require
a KIP though). – Even if we re-use the existing config, not 100% sure if we
would need a KIP or not?
> Allow to configure max memory for restore path
> ----------------------------------------------
>
> Key: KAFKA-19992
> URL: https://issues.apache.org/jira/browse/KAFKA-19992
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Muralidhar Basani
> Priority: Major
> Labels: needs-kip
>
> Kafka Streams allows users to configure many different memory limits, to
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The
> `StoreChangelogReader` fetches a batch of message, and buffers them inside
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would
> require a KIP though). – Even if we re-use the existing config, not 100% sure
> if we would need a KIP or not?
> KIP-1325:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
