[
https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419978#comment-15419978
]
ASF GitHub Bot commented on FLINK-4340:
---------------------------------------
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@StephanEwen you are right. But in specific situation, we may need some
temporary compromise do make the system work well, and then remove the
compromised points latter as soon as possible .
I think both approaches have shortcomes. When the state is large such as
millions of KVs per db, full async approach can do the full async backup, but
will cost a lot of time to restore which may be intolerable while doing
fail-over in production. So I think it is necessary to have both, and the full
async can be the default option.
Considering that there is no really perfect solution yet, I think It is OK
to remove the semi-async way right now to avoid blocking the job of key group
but need to reintroduce a better solution latter soon if you agree that rocksdb
is quite a good choice of statebackend in large state situations.
Regrading to the overhead of memory in different dbs. Rocksdb can share the
same block cache for different db instance but I don't know how to reduce the
cost of memtables which is also a problem existed in current solution that
allowing to store different stats in a single db using column families since
memtables of column families are also separated.
> Remove RocksDB Semi-Async Checkpoint Mode
> -----------------------------------------
>
> Key: FLINK-4340
> URL: https://issues.apache.org/jira/browse/FLINK-4340
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Affects Versions: 1.1.0
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
>
> This seems to be causing to many problems and is also incompatible with the
> upcoming key-group/sharding changes that will allow rescaling of keyed state.
> Once this is done we can also close FLINK-4228.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
