One thing I want to add: If you use window-operations, windows are kept until there retention time expires. Thus, reducing the retention time, should decrease the memory RocksDB needs to preserve windows.
See http://docs.confluent.io/3.0.0/streams/developer-guide.html?highlight=retention#windowing-a-stream You can set the retention time via > Windows#until(long duration) Of course, this is a trade-off between memory-usage and the ability to process late arriving data. As long as windows are not expired, they can get updated with late data. If retention time passed, late arrive records gets dropped. See http://docs.confluent.io/3.0.0/streams/concepts.html?highlight=late%20arriving#windowing -Matthias On 06/29/2016 05:49 PM, Eno Thereska wrote: > Hi Avi, > > These are internal files to RockDb. Depending on your load in the system I > suppose they could contain quite a bit of data. How large was the load in the > system these past two weeks so we can calibrate? Otherwise I'm not sure if > 1-2GB is a lot or not (sounds like not that big to make the disk full, was > there something else as well that ate up space?) > > Thanks > Eno > > >> On 29 Jun 2016, at 16:33, Avi Flax <avi.f...@parkassist.com> wrote: >> >> Hello all, >> >> I’ve been working on a Streams app and so far it’s going quite well. I’ve >> had the app up and running in a staging environment for a few weeks, >> processing real data. >> >> Today I logged into the server to check in on some things, and I found the >> disk was full. >> >> I managed (with ncdu) to find that most of the unexpected usage was in: >> >> /tmp/kafka-streams/my_app/0_0/rocksdb/my_intermediate_aggregate_topic >> >> and specifically in files named LOG.old.{timestamp}. The files were 2.6 GB >> and 118 MB. Also, LOG was 1.1 GB. >> >> I checked the docs and searched the list (and Google) for info on Kafka >> Streams and/or RocksDB disk usage, cleanup, etc, but didn’t find much. >> >> So: >> >> * Is this normal, expected behavior? >> >> * Any recommendations or suggestions for dealing with this? >> >> Thanks! >> Avi >
signature.asc
Description: OpenPGP digital signature
