[ 
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127017#comment-18127017
 ] 

Karsten Stöckmann commented on KAFKA-19992:
-------------------------------------------

[~unknowntpo] Thanks a lot for getting back. I'm happy to share whatever helps 
resolving this issue (in case it's at hand, I'll dig around a bit)... - So:
h4. Versions

Kafka Streams {_}4.2.1{_}, Quarkus _3.38.2_
h4. How the topology is built
 # I'm using {{builder.build(properties)}} - and, yes, I'm setting 
{{{}topology.optimization=all{}}}.
 # As it seems, *no* changelog topics for tables read with 
{{{}builder.table(...){}}}.

h4. Kafka Streams configs

({{{}rocksdb.*{}}} configs are used to configure the mentioned 
{{{}BoundedRocksDBConfigSetter{}}})

 
{code:java}
    kafka-streams.acks=all
    kafka-streams.auto.offset.reset=earliest
    kafka-streams.cache.max.bytes.buffering=10485760
    kafka-streams.commit.interval.ms=5000
    kafka-streams.consumer.max.partition.fetch.bytes=5242880
    kafka-streams.ensure.explicit.internal.resource.naming=true
    kafka-streams.max.warmup.replicas=2
    kafka-streams.metrics.recording.level=DEBUG
    kafka-streams.num.standby.replicas=1
    kafka-streams.num.stream.threads=1
    kafka-streams.producer.max.request.size=5242880
    
kafka-streams.rocksdb.config.setter=<redacted>.service.kstreams.config.BoundedRocksDBConfigSetter
    kafka-streams.statestore.cache.max.bytes=10485760
    kafka-streams.topic.compression.type=lz4
    kafka-streams.topic.delete.retention.ms=86400000
    kafka-streams.topic.max.compaction.lag.ms=86400000
    kafka-streams.topic.segment.ms=43200000
    kafka-streams.topology.optimization=all
    rocksdb.block.size.bytes=65536
    rocksdb.compaction.universal.max.size.amplification.percent=25
    rocksdb.compression.type=LZ4_COMPRESSION
    rocksdb.index.filter.block.ratio=0.2
    rocksdb.info.log.level=ERROR_LEVEL
    rocksdb.max.background.jobs=2
    rocksdb.max.open.files=1000
    rocksdb.max.subcompactions=1
    rocksdb.metadata.block.size.bytes=4096
    rocksdb.partition.filters=true
    rocksdb.pin.l0.filter.and.index.blocks.in.cache=false
    rocksdb.pin.top.level.index.and.filter=false
    rocksdb.skip.check.sst.file.sizes.on.db.open=true
    rocksdb.skip.stats.update.on.db.open=true
    rocksdb.strict.capacity.limit=true
    rocksdb.total.off.heap.memory=1GiB
    rocksdb.use.partitioned.index.filters=true {code}
h4. Size of the backfill
 # 18 partitions, 16 application instances while backfilling (2 in normal 
operation), typical record size - very hard to tell, I'd say roughly 500b, but 
may be significantly bigger.
 # Memory configuration:

{code:java}
      jvm: >-
        -Xms9G
        -Xmx9G
        -XX:G1HeapRegionSize=4M
      resources:
        limits:
          memory: "12Gi"
        requests:
          cpu: "200m"
          memory: "12Gi"{code}

h4. The OutOfMemoryError itself

This is the hard part. I doubt I've got any error information or stack traces 
left; the last time the issues occurred is rather months ago (i.e., our last 
backfilling situation), and our logs have 30d retention time. As far as I 
recall, the errors came up in a restore consumer (heap dumps suggested the 
same). I'll go digging... - I'll report back in case there's anything of 
interest left... (maybe there's something in related discussions in the 
Confluent Slack community. I'll go have a look).

 

 

> Allow to configure max memory for restore path
> ----------------------------------------------
>
>                 Key: KAFKA-19992
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19992
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Muralidhar Basani
>            Priority: Major
>              Labels: kip
>             Fix For: 4.5.0
>
>
> Kafka Streams allows users to configure many different memory limits, to 
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The 
> `StoreChangelogReader` fetches a batch of message, and buffers them inside 
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would 
> require a KIP though). – Even if we re-use the existing config, not 100% sure 
> if we would need a KIP or not?
> KIP-1325: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to