[
https://issues.apache.org/jira/browse/CAMEL-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-27037 started by shashank.
----------------------------------------
> camel-zookeeper-master - a group closed while the client is disconnected, or
> while it waits for the answer of a create, leaves its znode in the group as
> long as the session lives
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27037
> URL: https://issues.apache.org/jira/browse/CAMEL-27037
> Project: Camel
> Issue Type: Bug
> Components: camel-zookeeper-master
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found by the zookeeper-master DST world of CAMEL-26974 (PR #30137, branch
> {{camel-zookeeper-master-dst-world}}), in two cases its default set leaves
> out: with {{-Ddst.zookeeper.stopWhileSuspended=true}} 7 of 5000 seeds fail,
> and with {{-Ddst.zookeeper.stopDuringCreate=true}} 100 of the first 1189
> seeds fail (the run stops at 100), on "once the faults cleared, the master
> runs its consumer" and "no znode of this node is left in the group after the
> final stop". Reproduced with a plain unit test (below) against a ZooKeeper
> server in the JVM, outside the DST harness.
> h3. What happens
> When the master consumer stops or the route of the {{MasterRoutePolicy}} is
> removed, its {{ZooKeeperGroup}} is closed: it interrupts its thread and
> deletes its ephemeral znode ({{doUpdate(null)}}). The CuratorFramework is
> often shared and outlives the group (the client of the component, or the
> {{curator}} bean), so its session, and every ephemeral znode of the session,
> lives on. Two cases leave the znode in the group:
> # *Closed while the client is disconnected.* {{doUpdate(null)}} deletes the
> znode only {{if (isConnected())}}, and then forgets its path ({{id = null}}).
> When the client reconnects to the same session (a connection loss shorter
> than the session timeout), the znode is still there.
> # *Closed while the thread of the group waits for the answer of a create.*
> The interrupt ends the wait, but the create was sent and is applied by the
> server; the group never records the path, and {{doUpdate(null)}} only logs
> "Ephemeral node could be created in the registry, but ZooKeeper group didn't
> record its id".
> The znode stays in the group as long as the session lives, as a member of its
> own (the container id is new for each join, unless the {{runtime.id}} system
> property is set). While it has the lowest sequence it is the master of the
> group, so no node runs the route, this node included after its route started
> again (it joins behind the left-over znode).
> h3. Expected
> ZooKeeper removes an ephemeral znode only when the session that created it
> ends (Programmer's Guide, "Ephemeral Nodes"), so a znode a group leaves
> behind on a live session must be deleted by the client. Curator's
> {{DeleteBuilder.guaranteed()}} exists for this case: Curator records a failed
> delete and will "attempt to delete them in the background until successful",
> as long as the CuratorFramework is open. The znodes of a group carry its uuid
> ({{NodeState.uuid}}), which {{prunePartialState}} already uses to delete the
> duplicate of a create whose answer was lost.
> h3. Reproduction
> New {{ZooKeeperGroupCloseTest}} (a {{ZooKeeperServer}} in the JVM, as
> {{MasterRoutePolicyRouteLifecycleTest}}, and a CuratorFramework with a 30
> second session):
> * {{groupClosedWhileDisconnectedDeletesItsZnodeOnceReconnected}}: a group
> joins, the server stops, the group is closed while disconnected, the server
> starts again on the same data directory and the client reconnects to the same
> session. On main the znode is still in the group 20 seconds later.
> * {{groupClosedDuringACreateDeletesTheZnode}}: the client applies the create
> and then waits for the answer until the thread is interrupted (as the
> ZooKeeper client does when a close interrupts it). On main the znode is still
> in the group after the close.
> * {{groupClosedDuringACreateBeforeItKnowsItIsConnectedAgainDeletesTheZnode}}:
> the same, with the group told that the connection was lost and not yet that
> it is back. On main the znode is still in the group after the close.
> h3. Fix
> In {{ZooKeeperGroup.doUpdate(null)}}:
> * when the client is disconnected, the znode is deleted with
> {{delete().guaranteed().inBackground()}}: the close does not wait for the
> connection, and Curator deletes the znode once the client is connected again
> (if the session expired first, ZooKeeper removed it already). When connected,
> the foreground guaranteed delete stays as it is.
> * when a create did not answer and the client is connected (the client
> itself, as the group may not have been told yet that it is connected again:
> the close removes the listener of the group first), the group reads the
> children of its path and deletes those with its uuid (the reads come after
> the create in the order of the session, so they see the znode if the create
> was applied). While the client is disconnected it still only logs (not
> covered: the answer of a create lost with the connection, and the close
> before the reconnect).
> With the fix merged into the world of CAMEL-26974 and two test-only changes
> to its fake (a guaranteed delete in the background is modelled, and an
> interrupted retry sleep ends the retries of the client, as Curator's
> {{SleepingRetry}} does), {{stopDuringCreate}} and {{stopWhileSuspended}} pass
> 5000 of 5000 seeds each, and the default set 5000 of 5000 (also on main with
> the changed fake). Branch
> {{camel-zookeeper-master-group-close-leaves-znode-v2}}.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)