[
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127675#comment-18127675
]
Eric Chang commented on KAFKA-19992:
------------------------------------
[~stoeckmk] Thanks a lot, this is very helpful. With
{{topology.optimization=all}}, {{num.standby.replicas=1}} and
{{StoreChangelogReader#changelogs}} taking most of the heap, it looks like the
same issue as KAFKA-21276.
Until a fix is released, I think these two options could help:
* *Do not use {{reuse.ktable.source.topics}}.* Each table then gets its own
changelog topic, and a standby only fetches what the active task has already
written there. In my test with a Debezium setup and {{-Xmx1g}}, the standby ran
out of memory in every run with {{topology.optimization=all}}, and never with
no optimization. You can probably keep the other two optimizations, for example
{{topology.optimization=merge.repartition.topics,single.store.self.join}},
since only {{reuse.ktable.source.topics}} makes a table use its input topic as
the changelog. This would also change less of the topology than {{none}}. (I
only tested {{all}} against no optimization, not this setting.) Two things to
be careful about:
** This changes the topology: Streams creates new changelog topics for these
tables, and they start empty. I have not tested switching an existing
application, but I expect it would need an application reset (or a new
{{application.id}}) and a reprocessing of the input, so it may fit best before
a backfill.
** It costs one more changelog topic per table (more storage and write traffic).
* *Set {{num.standby.replicas=0}} during a backfill*, and set it back after the
backfill. Only standby tasks seem to be affected. The cost is that without
standby tasks, a failed instance's tasks have to restore their state from
scratch on another instance.
If you get a chance to try one of these, it would be great if you could share
whether it helps.
> Allow to configure max memory for restore path
> ----------------------------------------------
>
> Key: KAFKA-19992
> URL: https://issues.apache.org/jira/browse/KAFKA-19992
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Muralidhar Basani
> Priority: Major
> Labels: kip
> Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The
> `StoreChangelogReader` fetches a batch of message, and buffers them inside
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would
> require a KIP though). – Even if we re-use the existing config, not 100% sure
> if we would need a KIP or not?
> KIP-1325:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)