Hi fellow Zookeepers, We are using Zookeeper C-client and server v3.5.1-alpha. We are observing the following sequence of events:-
Client 1 connected to leader. Client 1 creates ephemeral node /foo/bar Client 2 connected to follower Client 1 exits ----20s passes, which is configured as ephemeral node timeout-- Client 2 queries for children of /foo, gets an empty list Client 2 tries to remove ZNode /foo, gets an error ZNOTEMPTY It looks like cleanup of the ephemeral node /foo/bar was broadcast to follower but not committed on the leader. However, when Client 2 queried for children on the follower, this node was considered as deleted. However when Client 2 issued a delete for /foo, this operation failed at the leader due to /foo/bar not being fully cleaned up. Log messages are pasted below:- *Client 1:* Aug 18 23:27:56 INFO: Token 1000005ef complete on server [128.0.0.4:61887], sessionId=0x4000001ac5700ae, negotiated timeout=200 Aug 19 14:17:15 INFO: Create /system/nodes/re0/apps/crd-14104870620583660699/zkid returned 0 *Client 2:* Aug 18 23:27:56 INFO: Reconfiguring server at the client 128.0.0.4:61887, 128.0.0.5:61817,localhost:1111 Aug 19 14:18:35 DEBUG: Remove /system/nodes/re0/apps/crd-14104870620583660699 returned -111 *ZK Leader:* 2016-08-19 14:18:35,184 [myid:2147483652] - INFO [ProcessThread(sid:2147483652 cport:-1):: PrepRequestProcessor@649] - Processed session termination for sessionid: 0x4000001ac5700ae 2016-08-19 14:18:35,206 [myid:2147483652] - INFO [ProcessThread(sid:2147483652 cport:-1)::PrepRequestProcessor@843] - Got user-level KeeperException when processing sessionid:0x4000001ac57 0032 type:delete cxid:0x57b650fa zxid:0x100004ea9 txntype:-1 reqpath:n/a Error Path:/system/nodes/re0/apps/crd-14104870620583660699/zkid Error:KeeperErrorCode = Directory not empty for /sys tem/nodes/re0/apps/crd-14104870620583660699/zkid Based on my reading of the Zookeeper book(3 ed. p 169), this should not happen:- "To guarantee that the order of execution is preserved, CommitRequestProcessor stalls the processing of pending requests once it receives a write request. This means that any read requests that were received after a write request will be blocked until the write request passes the CommitRequestProcessor." Can someone explain this to me/correct my understanding? Thanks and regards, Shreyas
