[ 
https://issues.apache.org/jira/browse/KAFKA-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383778#comment-16383778
 ] 

Guozhang Wang edited comment on KAFKA-6603 at 3/2/18 4:30 PM:
--------------------------------------------------------------

Hi Igor, what's the difference between "my code" and "my pull request"? My pull 
request has been merged to trunk so if you compiled from trunk directly, that 
"code" should have the same effect as my pull request?

EDIT: never mind, I think I understand your explanation now, you were not using 
DSL but was writing on PAPI layer and access the underlying store directly, 
right? In that case yes the pull request would not help you, and you have to do 
the similar change as in the PR manually.


was (Author: guozhang):
Hi Igor, what's the difference between "my code" and "my pull request"? My pull 
request has been merged to trunk so if you compiled from trunk directly, that 
"code" should have the same effect as my pull request?

> Kafka streams off heap memory usage does not match expected values from 
> configuration
> -------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6603
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6603
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 1.0.0
>            Reporter: Igor Calabria
>            Priority: Minor
>
> Hi, I have a simple aggregation pipeline that's backed by the default state 
> store(rocksdb). The pipeline works fine except that off heap the memory usage 
> is way higher than expected. Following the 
> [documention|https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config]
>  has some effect(memory usage is reduced) but the values don't match at all. 
> The java process is set to run with just `-Xmx300m -Xms300m`  and rocksdb 
> config looks like this
> {code:java}
> tableConfig.setCacheIndexAndFilterBlocks(true);
> tableConfig.setBlockCacheSize(1048576); //1MB
> tableConfig.setBlockSize(16 * 1024); // 16KB
> options.setTableFormatConfig(tableConfig);
> options.setMaxWriteBufferNumber(2);
> options.setWriteBufferSize(8 * 1024); // 8KB{code}
> To estimate memory usage, I'm using this formula  
> {noformat}
> (block_cache_size + write_buffer_size * write_buffer_number) * segments * 
> partitions{noformat}
> Since my topic has 25 partitions with 3 segments each(it's a windowed store), 
> off heap memory usage should be about 76MB. What I'm seeing in production is 
> upwards of 300MB, even taking in consideration  extra overhead from rocksdb 
> compaction threads, this seems a bit high (especially when the disk usage for 
> all files is just 1GB) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to