StefanRRichter commented on a change in pull request #7163: [FLINK-10471][State
TTL] State TTL cleanup using RocksDb compaction filter
URL: https://github.com/apache/flink/pull/7163#discussion_r251839757
##########
File path: docs/dev/stream/state/state.md
##########
@@ -394,6 +396,49 @@ val ttlConfig = StateTtlConfig
This option is not applicable for the incremental checkpointing in the RocksDB
state backend.
+##### Cleanup during RocksDB compaction
+
+If RocksDB state backend is used, another cleanup strategy is to activate
Flink specific compaction filter.
+RocksDB periodically runs asynchronous compactions to merge state updates and
reduce storage.
+Flink compaction filter checks expiration timestamp of state entries with TTL
+and excludes expired values.
+
+This feature is disabled by default. It has to be firstly activated for the
RocksDB backend
+by setting Flink configuration option
`state.backend.rocksdb.ttl.comaction.filter.enabled`
+or by calling `RocksDBStateBackend::enableTtlCompactionFilter` if a custom
RocksDB state backend is created for a job.
+Then any state with TTL can be configured to use the filter:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+import org.apache.flink.api.common.state.StateTtlConfig;
+
+StateTtlConfig ttlConfig = StateTtlConfig
+ .newBuilder(Time.seconds(1))
+ .cleanupInRocksdbCompactFilter()
+ .build();
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+import org.apache.flink.api.common.state.StateTtlConfig
+
+val ttlConfig = StateTtlCon fig
+ .newBuilder(Time.seconds(1))
+ .cleanupInRocksdbCompactFilter
+ .build
+{% endhighlight %}
+</div>
+</div>
+
+You can activate debug logs from the native code of RocksDB filter
+by activating debug level for `FlinkCompactionFilter`:
+
+`log4j.logger.org.rocksdb.FlinkCompactionFilter=DEBUG`
+
+**Note:** Calling of TTL filter during compaction slows it down.
Review comment:
Can we say something about how much it will slow compation down and why?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
