[
https://issues.apache.org/jira/browse/FLINK-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin updated FLINK-10471:
------------------------------------
Description:
In case of rocksdb backend, we can piggy back compaction using [custom
compaction filter|https://github.com/facebook/rocksdb/wiki/Compaction-Filter]
which checks our last access timestamp and discards expired values. It requires
contributing a C++ Flink TTL specific filter to Rocksdb, like for
[cassandra|https://github.com/facebook/rocksdb/blob/master/utilities/cassandra/cassandra_compaction_filter.cc].
At the moment RocksDB does not support compaction filter plugins (see [PR
discussion|https://github.com/facebook/rocksdb/pull/4463]), it is under
development.
Meanwhile, we can apply to strategies to enable this feature in Flink:
* Release and maintain a temporary fork of RocksDB for Flink:
[FRocksDB|https://github.com/dataArtisans/frocksdb] and merge [TTL
filter|https://github.com/dataArtisans/frocksdb/pull/1] into this fork (used in
Flink 1.8)
* Build C++ TTL filter separately, pack this C++ lib into its JNI java client
jar and load it in Flink additionally to vanila RocksDB (Flink RocksDB
extensions, planned for Flink 1.9)
The second strategy is more flexible in the long run.
> State TTL cleanup using RocksDb compaction filter
> -------------------------------------------------
>
> Key: FLINK-10471
> URL: https://issues.apache.org/jira/browse/FLINK-10471
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
> Affects Versions: 1.7.0
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of rocksdb backend, we can piggy back compaction using [custom
> compaction filter|https://github.com/facebook/rocksdb/wiki/Compaction-Filter]
> which checks our last access timestamp and discards expired values. It
> requires contributing a C++ Flink TTL specific filter to Rocksdb, like for
> [cassandra|https://github.com/facebook/rocksdb/blob/master/utilities/cassandra/cassandra_compaction_filter.cc].
> At the moment RocksDB does not support compaction filter plugins (see [PR
> discussion|https://github.com/facebook/rocksdb/pull/4463]), it is under
> development.
> Meanwhile, we can apply to strategies to enable this feature in Flink:
> * Release and maintain a temporary fork of RocksDB for Flink:
> [FRocksDB|https://github.com/dataArtisans/frocksdb] and merge [TTL
> filter|https://github.com/dataArtisans/frocksdb/pull/1] into this fork (used
> in Flink 1.8)
> * Build C++ TTL filter separately, pack this C++ lib into its JNI java
> client jar and load it in Flink additionally to vanila RocksDB (Flink RocksDB
> extensions, planned for Flink 1.9)
> The second strategy is more flexible in the long run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
