Re: Size of Checkpoints increasing with time

2018-10-24 Thread Kien Truong
Hi, Do you use incremental checkpoint ? RocksDB is an append-only DB, so you will experience the steady increase in state size until a compaction occurs and old values of keys are garbage-collected. However, the average state size should stabilize after a while, if the load doesn't change.

Size of Checkpoints increasing with time

2018-10-23 Thread Sameer W
Hi, We are using ValueState to maintain state. It is a pretty simple job with a keyBy operator on a stream and the subsequent map operator maintains state in a ValueState instance. The transaction load is in billion transactions per day. However the amount of state per key is a list of 18x6 long v