Re: Streams RocksDB State Store Disk Usage

I find this tuning guide in RocksDB quite useful, regarding your write / space amplifications. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide Guozhang On Thu, Jun 30, 2016 at 8:36 AM, Avi Flax wrote: > On Jun 29, 2016, at 22:44, Guozhang Wang

Re: Streams RocksDB State Store Disk Usage

On Jun 29, 2016, at 22:44, Guozhang Wang wrote: > > One way to mentally quantify your state store usage is to consider the > total key space in your reduceByKey() operator, and multiply by the average > key-value pair size. Then you need to consider the RocksDB write / space

Re: Streams RocksDB State Store Disk Usage

Hello Avi, One way to mentally quantify your state store usage is to consider the total key space in your reduceByKey() operator, and multiply by the average key-value pair size. Then you need to consider the RocksDB write / space amplification factor as well. Currently Kafka Streams hard-write

Re: Streams RocksDB State Store Disk Usage

On Jun 29, 2016, at 14:15, Matthias J. Sax wrote: > > If you use window-operations, windows are kept until there retention > time expires. Thus, reducing the retention time, should decrease the > memory RocksDB needs to preserve windows. Thanks Matthias, that makes sense

Re: Streams RocksDB State Store Disk Usage

On Jun 29, 2016, at 11:49, Eno Thereska wrote: > These are internal files to RockDb. Yeah, that makes sense. However, since Streams is encapsulating/employing RocksDB, in my view it’s Streams’ responsibility to configure RocksDB well with good defaults and/or at least

Re: Streams RocksDB State Store Disk Usage

One thing I want to add: If you use window-operations, windows are kept until there retention time expires. Thus, reducing the retention time, should decrease the memory RocksDB needs to preserve windows. See

Re: Streams RocksDB State Store Disk Usage

Hi Avi, These are internal files to RockDb. Depending on your load in the system I suppose they could contain quite a bit of data. How large was the load in the system these past two weeks so we can calibrate? Otherwise I'm not sure if 1-2GB is a lot or not (sounds like not that big to make

Streams RocksDB State Store Disk Usage

Hello all, I’ve been working on a Streams app and so far it’s going quite well. I’ve had the app up and running in a staging environment for a few weeks, processing real data. Today I logged into the server to check in on some things, and I found the disk was full. I managed (with ncdu) to