[ 
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126960#comment-18126960
 ] 

Eric Chang commented on KAFKA-19992:
------------------------------------

Hi [~stoeckmk],

Thanks for reporting this. I am helping [~muralibasani] with KIP-1325, and I am 
trying to find out which part of the restore path uses the memory in your case. 
Could you share a few details about your setup? Anything you can share helps, 
even if it is only part of it.

*Versions*
# Kafka Streams version, and Quarkus version.

*How the topology is built*
# In the Quarkus method that produces the {{Topology}}, do you return 
{{builder.build()}} or {{builder.build(props)}}? If you pass properties, does 
{{topology.optimization}} appear there?
# Do you see changelog topics named {{<application.id>-<store name>-changelog}} 
for the tables you read with {{streamsBuilder.table(...)}}?

*Kafka Streams configs* (from your {{application.properties}}, the 
{{kafka-streams.*}} part)
# {{num.standby.replicas}}, {{num.stream.threads}}, {{max.warmup.replicas}}, 
{{acceptable.recovery.lag}}
# {{processing.guarantee}}, {{commit.interval.ms}}
# {{statestore.cache.max.bytes}}, {{buffered.records.per.partition}}
# Any consumer overrides, in particular {{restore.consumer.*}}, 
{{max.poll.records}}, {{max.partition.fetch.bytes}}, {{fetch.max.bytes}}
# Any custom {{rocksdb.config.setter}}

*Size of the backfill*
# Number of partitions of the input topics, number of application instances, 
and the typical record size.
# JVM heap ({{-Xmx}}) and container memory limit of one instance.

*The OutOfMemoryError itself*
# When does it happen: on the first start of a new instance, while standby 
tasks are being updated, or when you scale out?
# Which thread throws it? The thread name in the stack trace (for example 
{{...-StreamThread-1}} or {{...-StateUpdater-1}}) would already tell us a lot.
# If you have a heap dump or a class histogram ({{jcmd <pid> 
GC.class_histogram}}), the top entries would be very helpful.

Thanks a lot!


> Allow to configure max memory for restore path
> ----------------------------------------------
>
>                 Key: KAFKA-19992
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19992
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Muralidhar Basani
>            Priority: Major
>              Labels: kip
>             Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to 
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The 
> `StoreChangelogReader` fetches a batch of message, and buffers them inside 
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would 
> require a KIP though). – Even if we re-use the existing config, not 100% sure 
> if we would need a KIP or not?
> KIP-1325: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to