[ https://issues.apache.org/jira/browse/KAFKA-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
A. Sophie Blee-Goldman resolved KAFKA-6603. ------------------------------------------- Resolution: Fixed > Kafka streams off heap memory usage does not match expected values from > configuration > ------------------------------------------------------------------------------------- > > Key: KAFKA-6603 > URL: https://issues.apache.org/jira/browse/KAFKA-6603 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 1.0.0 > Reporter: Igor Calabria > Priority: Minor > > Hi, I have a simple aggregation pipeline that's backed by the default state > store(rocksdb). The pipeline works fine except that off heap the memory usage > is way higher than expected. Following the > [documention|https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config] > has some effect(memory usage is reduced) but the values don't match at all. > The java process is set to run with just `-Xmx300m -Xms300m` and rocksdb > config looks like this > {code:java} > tableConfig.setCacheIndexAndFilterBlocks(true); > tableConfig.setBlockCacheSize(1048576); //1MB > tableConfig.setBlockSize(16 * 1024); // 16KB > options.setTableFormatConfig(tableConfig); > options.setMaxWriteBufferNumber(2); > options.setWriteBufferSize(8 * 1024); // 8KB{code} > To estimate memory usage, I'm using this formula > {noformat} > (block_cache_size + write_buffer_size * write_buffer_number) * segments * > partitions{noformat} > Since my topic has 25 partitions with 3 segments each(it's a windowed store), > off heap memory usage should be about 76MB. What I'm seeing in production is > upwards of 300MB, even taking in consideration extra overhead from rocksdb > compaction threads, this seems a bit high (especially when the disk usage for > all files is just 1GB) -- This message was sent by Atlassian Jira (v8.3.4#803005)