Igor Calabria created KAFKA-6603:
------------------------------------

             Summary: Kafka streams off heap memory usage does not match 
expected values from configuration
                 Key: KAFKA-6603
                 URL: https://issues.apache.org/jira/browse/KAFKA-6603
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 1.0.0
            Reporter: Igor Calabria


Hi, I have a simple aggregation pipeline that's backed by the default state 
store(rocksdb). The pipeline works fine except that off heap the memory usage 
is way higher than expected. Following the 
[documention|https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config]
 has some effect(memory usage is reduced) but the values don't match at all. 

The java process is set to run with just `-Xmx300m -Xms300m`  and rocksdb 
config looks like this
{code:java}
tableConfig.setCacheIndexAndFilterBlocks(true);
tableConfig.setBlockCacheSize(1048576); //1MB
tableConfig.setBlockSize(16 * 1024); // 16KB
options.setTableFormatConfig(tableConfig);
options.setMaxWriteBufferNumber(2);
options.setWriteBufferSize(8 * 1024); // 8KB{code}
To estimate memory usage, I'm using this formula  
{noformat}
(block_cache_size + write_buffer_size * write_buffer_number) * segments * 
partitions{noformat}
Since my topic has 25 partitions with 3 segments each(it's a windowed store), 
off heap memory usage should be about 76MB. What I'm seeing in production is 
upwards of 300MB, even taking in consideration  extra overhead from rocksdb 
compaction threads, this seems a bit high (especially when the disk usage for 
all files is just 1GB) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to