Dear Flink users and developers!
I start this discussion to collect feedback about maintaining a custom
RocksDB branch for Flink, if anyone sees any problems with this approach.
Are there people who already uses a custom RocksDB client build with
RocksDB state backend?
As you might already know, the community is trying currently to make some
improvements in RocksDB state backend which require a newer version of
RocksDB client and some Flink specific changes of its code base.
In particular:
- custom c++ compaction filter for background cleanup of expired state
entries with TTL [1]
- merge operator without separating commas for the list state [2]
The TTL compaction filter is too Flink specific to merge it into the
RocksDB repository.
The second one might take time to be merged and released in RocksDB.
The plug-ability of filter/merge operators in RocksDB would allow the code
to be kept on Flink side but it is currently under development and ETA is
not cleat at the moment.
To unblock releasing of related Flink features, we suggest to create and
maintain a branch of RocksDB which would include these changes, as it
happened already in the past. The RocksDB backend module can be built
against this branch. We can get back to normal RocksDB releases, once it
allows Flink code to be kept out of it.
Thanks,
Andrey
[1] https://issues.apache.org/jira/browse/FLINK-10471
[2] https://github.com/facebook/rocksdb/pull/4806
