Hi Jinkui, the file system state backend and the RocksDB state backend can be configured (and usually should be) such that they store their checkpoint data on a reliable storage system such as HDFS. Then you also have the reliability guarantees.
Of course, one can start adding more state backends to Flink. At some point in time there was the idea to write a Cassandra backed state backend [1], for example. Similarly, one could think about a HBase backed state backend. [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Cassandra-statebackend-td12690.html Cheers, Till On Wed, Nov 16, 2016 at 3:10 AM, shijinkui <shijin...@huawei.com> wrote: > Hi, All > > At present flink have three state backend: memory, file system, rocksdb. > MemoryStateBackend will tansform the snapshot to jobManager, 5MB limited > default. Even setting it bigger, that not suitable for very big state > storage. > HDFS can meet the reliability guarantee, but It's slow. File System and > RocksDB are fast, but they are have no reliability guarantee. > Three state backend all have no reliability guarantee. > > Can we have a Hbase state backend, providing reliability guarantee of > state snapshot? > For user, only new a HbaseStateBackend object, provide hbase parameter and > optimization configure. > Maybe Hbase or other distributed key-value storage is heavyweight storage, > we only use hbase client to read/write asynchronously. > > -Jinkui Shi >
