shashank created CAMEL-27037:
--------------------------------
Summary: camel-zookeeper-master - a group closed while the client
is disconnected, or while it waits for the answer of a create, leaves its znode
in the group as long as the session lives
Key: CAMEL-27037
URL: https://issues.apache.org/jira/browse/CAMEL-27037
Project: Camel
Issue Type: Bug
Components: camel-zookeeper-master
Reporter: shashank
Assignee: shashank
Found by the zookeeper-master DST world of CAMEL-26974 (PR #30137, branch
{{camel-zookeeper-master-dst-world}}), in two cases its default set leaves out:
with {{-Ddst.zookeeper.stopWhileSuspended=true}} 7 of 5000 seeds fail, and with
{{-Ddst.zookeeper.stopDuringCreate=true}} 100 of the first 1189 seeds fail (the
run stops at 100), on "once the faults cleared, the master runs its consumer"
and "no znode of this node is left in the group after the final stop".
Reproduced with a plain unit test (below) against a ZooKeeper server in the
JVM, outside the DST harness.
h3. What happens
When the master consumer stops or the route of the {{MasterRoutePolicy}} is
removed, its {{ZooKeeperGroup}} is closed: it interrupts its thread and deletes
its ephemeral znode ({{doUpdate(null)}}). The CuratorFramework is often shared
and outlives the group (the client of the component, or the {{curator}} bean),
so its session, and every ephemeral znode of the session, lives on. Two cases
leave the znode in the group:
# *Closed while the client is disconnected.* {{doUpdate(null)}} deletes the
znode only {{if (isConnected())}}, and then forgets its path ({{id = null}}).
When the client reconnects to the same session (a connection loss shorter than
the session timeout), the znode is still there.
# *Closed while the thread of the group waits for the answer of a create.* The
interrupt ends the wait, but the create was sent and is applied by the server;
the group never records the path, and {{doUpdate(null)}} only logs "Ephemeral
node could be created in the registry, but ZooKeeper group didn't record its
id".
The znode stays in the group as long as the session lives, as a member of its
own (the container id is new for each join, unless the {{runtime.id}} system
property is set). While it has the lowest sequence it is the master of the
group, so no node runs the route, this node included after its route started
again (it joins behind the left-over znode).
h3. Expected
ZooKeeper removes an ephemeral znode only when the session that created it ends
(Programmer's Guide, "Ephemeral Nodes"), so a znode a group leaves behind on a
live session must be deleted by the client. Curator's
{{DeleteBuilder.guaranteed()}} exists for this case: Curator records a failed
delete and will "attempt to delete them in the background until successful", as
long as the CuratorFramework is open. The znodes of a group carry its uuid
({{NodeState.uuid}}), which {{prunePartialState}} already uses to delete the
duplicate of a create whose answer was lost.
h3. Reproduction
New {{ZooKeeperGroupCloseTest}} (a {{ZooKeeperServer}} in the JVM, as
{{MasterRoutePolicyRouteLifecycleTest}}, and a CuratorFramework with a 30
second session):
* {{groupClosedWhileDisconnectedDeletesItsZnodeOnceReconnected}}: a group
joins, the server stops, the group is closed while disconnected, the server
starts again on the same data directory and the client reconnects to the same
session. On main the znode is still in the group 20 seconds later.
* {{groupClosedDuringACreateDeletesTheZnode}}: the client applies the create
and then waits for the answer until the thread is interrupted (as the ZooKeeper
client does when a close interrupts it). On main the znode is still in the
group after the close.
* {{groupClosedDuringACreateBeforeItKnowsItIsConnectedAgainDeletesTheZnode}}:
the same, with the group told that the connection was lost and not yet that it
is back. On main the znode is still in the group after the close.
h3. Fix
In {{ZooKeeperGroup.doUpdate(null)}}:
* when the client is disconnected, the znode is deleted with
{{delete().guaranteed().inBackground()}}: the close does not wait for the
connection, and Curator deletes the znode once the client is connected again
(if the session expired first, ZooKeeper removed it already). When connected,
the foreground guaranteed delete stays as it is.
* when a create did not answer and the client is connected (the client itself,
as the group may not have been told yet that it is connected again: the close
removes the listener of the group first), the group reads the children of its
path and deletes those with its uuid (the reads come after the create in the
order of the session, so they see the znode if the create was applied). While
the client is disconnected it still only logs (not covered: the answer of a
create lost with the connection, and the close before the reconnect).
With the fix merged into the world of CAMEL-26974 and two test-only changes to
its fake (a guaranteed delete in the background is modelled, and an interrupted
retry sleep ends the retries of the client, as Curator's {{SleepingRetry}}
does), {{stopDuringCreate}} and {{stopWhileSuspended}} pass 5000 of 5000 seeds
each, and the default set 5000 of 5000 (also on main with the changed fake).
Branch {{camel-zookeeper-master-group-close-leaves-znode-v2}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)