[
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127017#comment-18127017
]
Karsten Stöckmann commented on KAFKA-19992:
-------------------------------------------
[~unknowntpo] Thanks a lot for getting back. I'm happy to share whatever helps
resolving this issue (in case it's at hand, I'll dig around a bit)... - So:
h4. Versions
Kafka Streams {_}4.2.1{_}, Quarkus _3.38.2_
h4. How the topology is built
# I'm using {{builder.build(properties)}} - and, yes, I'm setting
{{{}topology.optimization=all{}}}.
# As it seems, *no* changelog topics for tables read with
{{{}builder.table(...){}}}.
h4. Kafka Streams configs
({{{}rocksdb.*{}}} configs are used to configure the mentioned
{{{}BoundedRocksDBConfigSetter{}}})
{code:java}
kafka-streams.acks=all
kafka-streams.auto.offset.reset=earliest
kafka-streams.cache.max.bytes.buffering=10485760
kafka-streams.commit.interval.ms=5000
kafka-streams.consumer.max.partition.fetch.bytes=5242880
kafka-streams.ensure.explicit.internal.resource.naming=true
kafka-streams.max.warmup.replicas=2
kafka-streams.metrics.recording.level=DEBUG
kafka-streams.num.standby.replicas=1
kafka-streams.num.stream.threads=1
kafka-streams.producer.max.request.size=5242880
kafka-streams.rocksdb.config.setter=<redacted>.service.kstreams.config.BoundedRocksDBConfigSetter
kafka-streams.statestore.cache.max.bytes=10485760
kafka-streams.topic.compression.type=lz4
kafka-streams.topic.delete.retention.ms=86400000
kafka-streams.topic.max.compaction.lag.ms=86400000
kafka-streams.topic.segment.ms=43200000
kafka-streams.topology.optimization=all
rocksdb.block.size.bytes=65536
rocksdb.compaction.universal.max.size.amplification.percent=25
rocksdb.compression.type=LZ4_COMPRESSION
rocksdb.index.filter.block.ratio=0.2
rocksdb.info.log.level=ERROR_LEVEL
rocksdb.max.background.jobs=2
rocksdb.max.open.files=1000
rocksdb.max.subcompactions=1
rocksdb.metadata.block.size.bytes=4096
rocksdb.partition.filters=true
rocksdb.pin.l0.filter.and.index.blocks.in.cache=false
rocksdb.pin.top.level.index.and.filter=false
rocksdb.skip.check.sst.file.sizes.on.db.open=true
rocksdb.skip.stats.update.on.db.open=true
rocksdb.strict.capacity.limit=true
rocksdb.total.off.heap.memory=1GiB
rocksdb.use.partitioned.index.filters=true {code}
h4. Size of the backfill
# 18 partitions, 16 application instances while backfilling (2 in normal
operation), typical record size - very hard to tell, I'd say roughly 500b, but
may be significantly bigger.
# Memory configuration:
{code:java}
jvm: >-
-Xms9G
-Xmx9G
-XX:G1HeapRegionSize=4M
resources:
limits:
memory: "12Gi"
requests:
cpu: "200m"
memory: "12Gi"{code}
h4. The OutOfMemoryError itself
This is the hard part. I doubt I've got any error information or stack traces
left; the last time the issues occurred is rather months ago (i.e., our last
backfilling situation), and our logs have 30d retention time. As far as I
recall, the errors came up in a restore consumer (heap dumps suggested the
same). I'll go digging... - I'll report back in case there's anything of
interest left... (maybe there's something in related discussions in the
Confluent Slack community. I'll go have a look).
> Allow to configure max memory for restore path
> ----------------------------------------------
>
> Key: KAFKA-19992
> URL: https://issues.apache.org/jira/browse/KAFKA-19992
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Muralidhar Basani
> Priority: Major
> Labels: kip
> Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The
> `StoreChangelogReader` fetches a batch of message, and buffers them inside
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would
> require a KIP though). – Even if we re-use the existing config, not 100% sure
> if we would need a KIP or not?
> KIP-1325:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)