MalcolmSanders created FLINK-11809:
--------------------------------------
Summary: Implement etcd based StateHandleStore
Key: FLINK-11809
URL: https://issues.apache.org/jira/browse/FLINK-11809
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: MalcolmSanders
Implement StateHandleStore using jetcd.
Previously ZooKeeperStateHandleStore stores data in a dfs file while records
its metadata to a zookeeper node in order to keep the data size of a zookeeper
node small. EtcdStateHandleStore should work in the same way while there is a
corner case that should be carefully dealt with using etcd.
As described in FLINK-6612, the ZooKeeperStateHandleStore does not guard
against concurrent delete operations which could happen in case of a lost
leadership and a new leadership grant. The problem is that checkpoint nodes can
get deleted even after they have been recovered by another
ZooKeeperCompletedCheckpointStore. This corrupts the recovered checkpoint and
thwarts future recoveries. In order to guard against deletions of ZooKeeper
nodes which are still being used by a different ZooKeeperStateHandleStore, a
locking mechanism has been introduced to make sure that zookeeper nodes are
allowed to be deleted only after all ZooKeeperStateHandleStores have released
their lock. The locking mechanism is implemented via ephemeral child nodes of
the respective ZooKeeper node. Whenever a ZooKeeperStateHandleStore wants to
lock a ZNode, thus, protecting it from being deleted, it creates an ephemeral
child node. The node's name is unique to the ZooKeeperStateHandleStore
instance. The delete operations will then only delete the node if it does not
have any children associated. In order to guard against orphaned lock nodes,
they are created as ephemeral nodes. This means that they will be deleted by
ZooKeeper once the connection of the ZooKeeper client which created the node
timed out.
The solution leverages that a zookeeper directory cannot be deleted if it still
has child nodes and a ephemeral node will be deleted by zookeeper server once
the corresponding client is disconnected. Since etcd doesn’t have hierarchical
structure like zookeeper, there is no actual relations between a path and its
so-called parent directory. Suppose we create a persistent key-value to store
metadata and then create an ephemeral key-value using previous key as the
prefix. Once the client disconnects with etcd server, the ephemeral key-value
pair will be deleted from etcd server while there’ll be no effect on its parent
key-value pair.
My proposal to tackle this case is illustrated in Part 6 Implementation of etcd
based StateHandleStore in [design
doc|https://docs.google.com/document/d/12-gIZDuT4IOWG7gmwSqNFsOHuGlkdRHge0ahJ7M311Y/edit#heading=h.sqkj9zjvgicu].
Any comments or suggestions will be appreciated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
