[
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126960#comment-18126960
]
Eric Chang commented on KAFKA-19992:
------------------------------------
Hi [~stoeckmk],
Thanks for reporting this. I am helping [~muralibasani] with KIP-1325, and I am
trying to find out which part of the restore path uses the memory in your case.
Could you share a few details about your setup? Anything you can share helps,
even if it is only part of it.
*Versions*
# Kafka Streams version, and Quarkus version.
*How the topology is built*
# In the Quarkus method that produces the {{Topology}}, do you return
{{builder.build()}} or {{builder.build(props)}}? If you pass properties, does
{{topology.optimization}} appear there?
# Do you see changelog topics named {{<application.id>-<store name>-changelog}}
for the tables you read with {{streamsBuilder.table(...)}}?
*Kafka Streams configs* (from your {{application.properties}}, the
{{kafka-streams.*}} part)
# {{num.standby.replicas}}, {{num.stream.threads}}, {{max.warmup.replicas}},
{{acceptable.recovery.lag}}
# {{processing.guarantee}}, {{commit.interval.ms}}
# {{statestore.cache.max.bytes}}, {{buffered.records.per.partition}}
# Any consumer overrides, in particular {{restore.consumer.*}},
{{max.poll.records}}, {{max.partition.fetch.bytes}}, {{fetch.max.bytes}}
# Any custom {{rocksdb.config.setter}}
*Size of the backfill*
# Number of partitions of the input topics, number of application instances,
and the typical record size.
# JVM heap ({{-Xmx}}) and container memory limit of one instance.
*The OutOfMemoryError itself*
# When does it happen: on the first start of a new instance, while standby
tasks are being updated, or when you scale out?
# Which thread throws it? The thread name in the stack trace (for example
{{...-StreamThread-1}} or {{...-StateUpdater-1}}) would already tell us a lot.
# If you have a heap dump or a class histogram ({{jcmd <pid>
GC.class_histogram}}), the top entries would be very helpful.
Thanks a lot!
> Allow to configure max memory for restore path
> ----------------------------------------------
>
> Key: KAFKA-19992
> URL: https://issues.apache.org/jira/browse/KAFKA-19992
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Muralidhar Basani
> Priority: Major
> Labels: kip
> Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The
> `StoreChangelogReader` fetches a batch of message, and buffers them inside
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would
> require a KIP though). – Even if we re-use the existing config, not 100% sure
> if we would need a KIP or not?
> KIP-1325:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)