Re: Question about RocksDBStateBackend Compaction Filter state cleanup

2020-03-17 Thread Andrey Zagrebin
Hi Lake, When the Flink doc mentions a state entry in RocksDB, we mean one key/value pair stored by user code over any keyed state API (keyed context in keyed operators obtained e.g. from keyBy() transformation). In case of map or list, the doc means map key/value and list element. - value/aggreg

Re: Question about RocksDBStateBackend Compaction Filter state cleanup

2020-03-17 Thread Yun Tang
Hi Lake Flink leverage RocksDB's background compaction mechanism to filter out-of-TTL entries (by comparing with current timestamp provided from RocksDB's time_provider) to not let them stay in newly compacted data. This would iterator over data entries with FlinkCompactionFilter::FilterV2 [1],