shashank created CAMEL-27037:
--------------------------------

             Summary: camel-zookeeper-master - a group closed while the client 
is disconnected, or while it waits for the answer of a create, leaves its znode 
in the group as long as the session lives
                 Key: CAMEL-27037
                 URL: https://issues.apache.org/jira/browse/CAMEL-27037
             Project: Camel
          Issue Type: Bug
          Components: camel-zookeeper-master
            Reporter: shashank
            Assignee: shashank


Found by the zookeeper-master DST world of CAMEL-26974 (PR #30137, branch 
{{camel-zookeeper-master-dst-world}}), in two cases its default set leaves out: 
with {{-Ddst.zookeeper.stopWhileSuspended=true}} 7 of 5000 seeds fail, and with 
{{-Ddst.zookeeper.stopDuringCreate=true}} 100 of the first 1189 seeds fail (the 
run stops at 100), on "once the faults cleared, the master runs its consumer" 
and "no znode of this node is left in the group after the final stop". 
Reproduced with a plain unit test (below) against a ZooKeeper server in the 
JVM, outside the DST harness.

h3. What happens

When the master consumer stops or the route of the {{MasterRoutePolicy}} is 
removed, its {{ZooKeeperGroup}} is closed: it interrupts its thread and deletes 
its ephemeral znode ({{doUpdate(null)}}). The CuratorFramework is often shared 
and outlives the group (the client of the component, or the {{curator}} bean), 
so its session, and every ephemeral znode of the session, lives on. Two cases 
leave the znode in the group:
# *Closed while the client is disconnected.* {{doUpdate(null)}} deletes the 
znode only {{if (isConnected())}}, and then forgets its path ({{id = null}}). 
When the client reconnects to the same session (a connection loss shorter than 
the session timeout), the znode is still there.
# *Closed while the thread of the group waits for the answer of a create.* The 
interrupt ends the wait, but the create was sent and is applied by the server; 
the group never records the path, and {{doUpdate(null)}} only logs "Ephemeral 
node could be created in the registry, but ZooKeeper group didn't record its 
id".

The znode stays in the group as long as the session lives, as a member of its 
own (the container id is new for each join, unless the {{runtime.id}} system 
property is set). While it has the lowest sequence it is the master of the 
group, so no node runs the route, this node included after its route started 
again (it joins behind the left-over znode).

h3. Expected

ZooKeeper removes an ephemeral znode only when the session that created it ends 
(Programmer's Guide, "Ephemeral Nodes"), so a znode a group leaves behind on a 
live session must be deleted by the client. Curator's 
{{DeleteBuilder.guaranteed()}} exists for this case: Curator records a failed 
delete and will "attempt to delete them in the background until successful", as 
long as the CuratorFramework is open. The znodes of a group carry its uuid 
({{NodeState.uuid}}), which {{prunePartialState}} already uses to delete the 
duplicate of a create whose answer was lost.

h3. Reproduction

New {{ZooKeeperGroupCloseTest}} (a {{ZooKeeperServer}} in the JVM, as 
{{MasterRoutePolicyRouteLifecycleTest}}, and a CuratorFramework with a 30 
second session):
* {{groupClosedWhileDisconnectedDeletesItsZnodeOnceReconnected}}: a group 
joins, the server stops, the group is closed while disconnected, the server 
starts again on the same data directory and the client reconnects to the same 
session. On main the znode is still in the group 20 seconds later.
* {{groupClosedDuringACreateDeletesTheZnode}}: the client applies the create 
and then waits for the answer until the thread is interrupted (as the ZooKeeper 
client does when a close interrupts it). On main the znode is still in the 
group after the close.
* {{groupClosedDuringACreateBeforeItKnowsItIsConnectedAgainDeletesTheZnode}}: 
the same, with the group told that the connection was lost and not yet that it 
is back. On main the znode is still in the group after the close.

h3. Fix

In {{ZooKeeperGroup.doUpdate(null)}}:
* when the client is disconnected, the znode is deleted with 
{{delete().guaranteed().inBackground()}}: the close does not wait for the 
connection, and Curator deletes the znode once the client is connected again 
(if the session expired first, ZooKeeper removed it already). When connected, 
the foreground guaranteed delete stays as it is.
* when a create did not answer and the client is connected (the client itself, 
as the group may not have been told yet that it is connected again: the close 
removes the listener of the group first), the group reads the children of its 
path and deletes those with its uuid (the reads come after the create in the 
order of the session, so they see the znode if the create was applied). While 
the client is disconnected it still only logs (not covered: the answer of a 
create lost with the connection, and the close before the reconnect).

With the fix merged into the world of CAMEL-26974 and two test-only changes to 
its fake (a guaranteed delete in the background is modelled, and an interrupted 
retry sleep ends the retries of the client, as Curator's {{SleepingRetry}} 
does), {{stopDuringCreate}} and {{stopWhileSuspended}} pass 5000 of 5000 seeds 
each, and the default set 5000 of 5000 (also on main with the changed fake). 
Branch {{camel-zookeeper-master-group-close-leaves-znode-v2}}.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to