[ 
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127675#comment-18127675
 ] 

Eric Chang commented on KAFKA-19992:
------------------------------------

[~stoeckmk] Thanks a lot, this is very helpful. With 
{{topology.optimization=all}}, {{num.standby.replicas=1}} and 
{{StoreChangelogReader#changelogs}} taking most of the heap, it looks like the 
same issue as KAFKA-21276.

Until a fix is released, I think these two options could help:
* *Do not use {{reuse.ktable.source.topics}}.* Each table then gets its own 
changelog topic, and a standby only fetches what the active task has already 
written there. In my test with a Debezium setup and {{-Xmx1g}}, the standby ran 
out of memory in every run with {{topology.optimization=all}}, and never with 
no optimization. You can probably keep the other two optimizations, for example 
{{topology.optimization=merge.repartition.topics,single.store.self.join}}, 
since only {{reuse.ktable.source.topics}} makes a table use its input topic as 
the changelog. This would also change less of the topology than {{none}}. (I 
only tested {{all}} against no optimization, not this setting.) Two things to 
be careful about:
** This changes the topology: Streams creates new changelog topics for these 
tables, and they start empty. I have not tested switching an existing 
application, but I expect it would need an application reset (or a new 
{{application.id}}) and a reprocessing of the input, so it may fit best before 
a backfill.
** It costs one more changelog topic per table (more storage and write traffic).
* *Set {{num.standby.replicas=0}} during a backfill*, and set it back after the 
backfill. Only standby tasks seem to be affected. The cost is that without 
standby tasks, a failed instance's tasks have to restore their state from 
scratch on another instance.

If you get a chance to try one of these, it would be great if you could share 
whether it helps.


> Allow to configure max memory for restore path
> ----------------------------------------------
>
>                 Key: KAFKA-19992
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19992
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Muralidhar Basani
>            Priority: Major
>              Labels: kip
>             Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to 
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The 
> `StoreChangelogReader` fetches a batch of message, and buffers them inside 
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would 
> require a KIP though). – Even if we re-use the existing config, not 100% sure 
> if we would need a KIP or not?
> KIP-1325: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to