Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2345
@wenlong88 I was talking about this effort to enable key-group sharding in
Flink: https://issues.apache.org/jira/browse/FLINK-3755
With this is becomes necessary to checkpoint the keyed states in a way that
operators can restore the state for the key shard for which they are
responsible. The state backend therefore has to write the snapshot data such
that keys for different key-groups are separated. If we keep the approach of
simply checkpointing the whole RocksDB DB we don't get this property. In the
fully-async checkpointing mode we iterate over the keys for each key group and
write them separately from the keys of the other key groups. (logically
speaking. they can still go into the same file but we need an index that tells
us where the state for each key group resides.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
