[ 
https://issues.apache.org/jira/browse/CAMEL-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-27037 started by shashank.
----------------------------------------
> camel-zookeeper-master - a group closed while the client is disconnected, or 
> while it waits for the answer of a create, leaves its znode in the group as 
> long as the session lives
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27037
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27037
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-zookeeper-master
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> Found by the zookeeper-master DST world of CAMEL-26974 (PR #30137, branch 
> {{camel-zookeeper-master-dst-world}}), in two cases its default set leaves 
> out: with {{-Ddst.zookeeper.stopWhileSuspended=true}} 7 of 5000 seeds fail, 
> and with {{-Ddst.zookeeper.stopDuringCreate=true}} 100 of the first 1189 
> seeds fail (the run stops at 100), on "once the faults cleared, the master 
> runs its consumer" and "no znode of this node is left in the group after the 
> final stop". Reproduced with a plain unit test (below) against a ZooKeeper 
> server in the JVM, outside the DST harness.
> h3. What happens
> When the master consumer stops or the route of the {{MasterRoutePolicy}} is 
> removed, its {{ZooKeeperGroup}} is closed: it interrupts its thread and 
> deletes its ephemeral znode ({{doUpdate(null)}}). The CuratorFramework is 
> often shared and outlives the group (the client of the component, or the 
> {{curator}} bean), so its session, and every ephemeral znode of the session, 
> lives on. Two cases leave the znode in the group:
> # *Closed while the client is disconnected.* {{doUpdate(null)}} deletes the 
> znode only {{if (isConnected())}}, and then forgets its path ({{id = null}}). 
> When the client reconnects to the same session (a connection loss shorter 
> than the session timeout), the znode is still there.
> # *Closed while the thread of the group waits for the answer of a create.* 
> The interrupt ends the wait, but the create was sent and is applied by the 
> server; the group never records the path, and {{doUpdate(null)}} only logs 
> "Ephemeral node could be created in the registry, but ZooKeeper group didn't 
> record its id".
> The znode stays in the group as long as the session lives, as a member of its 
> own (the container id is new for each join, unless the {{runtime.id}} system 
> property is set). While it has the lowest sequence it is the master of the 
> group, so no node runs the route, this node included after its route started 
> again (it joins behind the left-over znode).
> h3. Expected
> ZooKeeper removes an ephemeral znode only when the session that created it 
> ends (Programmer's Guide, "Ephemeral Nodes"), so a znode a group leaves 
> behind on a live session must be deleted by the client. Curator's 
> {{DeleteBuilder.guaranteed()}} exists for this case: Curator records a failed 
> delete and will "attempt to delete them in the background until successful", 
> as long as the CuratorFramework is open. The znodes of a group carry its uuid 
> ({{NodeState.uuid}}), which {{prunePartialState}} already uses to delete the 
> duplicate of a create whose answer was lost.
> h3. Reproduction
> New {{ZooKeeperGroupCloseTest}} (a {{ZooKeeperServer}} in the JVM, as 
> {{MasterRoutePolicyRouteLifecycleTest}}, and a CuratorFramework with a 30 
> second session):
> * {{groupClosedWhileDisconnectedDeletesItsZnodeOnceReconnected}}: a group 
> joins, the server stops, the group is closed while disconnected, the server 
> starts again on the same data directory and the client reconnects to the same 
> session. On main the znode is still in the group 20 seconds later.
> * {{groupClosedDuringACreateDeletesTheZnode}}: the client applies the create 
> and then waits for the answer until the thread is interrupted (as the 
> ZooKeeper client does when a close interrupts it). On main the znode is still 
> in the group after the close.
> * {{groupClosedDuringACreateBeforeItKnowsItIsConnectedAgainDeletesTheZnode}}: 
> the same, with the group told that the connection was lost and not yet that 
> it is back. On main the znode is still in the group after the close.
> h3. Fix
> In {{ZooKeeperGroup.doUpdate(null)}}:
> * when the client is disconnected, the znode is deleted with 
> {{delete().guaranteed().inBackground()}}: the close does not wait for the 
> connection, and Curator deletes the znode once the client is connected again 
> (if the session expired first, ZooKeeper removed it already). When connected, 
> the foreground guaranteed delete stays as it is.
> * when a create did not answer and the client is connected (the client 
> itself, as the group may not have been told yet that it is connected again: 
> the close removes the listener of the group first), the group reads the 
> children of its path and deletes those with its uuid (the reads come after 
> the create in the order of the session, so they see the znode if the create 
> was applied). While the client is disconnected it still only logs (not 
> covered: the answer of a create lost with the connection, and the close 
> before the reconnect).
> With the fix merged into the world of CAMEL-26974 and two test-only changes 
> to its fake (a guaranteed delete in the background is modelled, and an 
> interrupted retry sleep ends the retries of the client, as Curator's 
> {{SleepingRetry}} does), {{stopDuringCreate}} and {{stopWhileSuspended}} pass 
> 5000 of 5000 seeds each, and the default set 5000 of 5000 (also on main with 
> the changed fake). Branch 
> {{camel-zookeeper-master-group-close-leaves-znode-v2}}.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to