[ 
https://issues.apache.org/jira/browse/CAMEL-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146097#comment-18146097
 ] 

shashank commented on CAMEL-27037:
----------------------------------

Case left open by the fix in https://github.com/apache/camel/pull/30249 (noted 
from the review):

A create of the group whose answer was lost (the connection dropped while the 
create was in flight), followed by a close of the group while the client is 
still disconnected. The group does not know the path of the znode, and while 
disconnected it cannot read the children of the group to find the znode by its 
uuid, so it logs a WARN and leaves it. If the create was applied and the client 
reconnects within the session timeout, that znode stays in the group as long as 
the session lives.

A fix would need the lookup by uuid to run once the client is connected again 
(as Curator does for a background guaranteed delete of a known path), or a 
protected create ({{withProtection()}}), whose node name carries an id that 
Curator can find after a lost answer. Not covered by #30249.

_Written with Claude Code on behalf of allthingssecurity._

> camel-zookeeper-master - a group closed while the client is disconnected, or 
> while it waits for the answer of a create, leaves its znode in the group as 
> long as the session lives
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27037
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27037
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-zookeeper-master
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> Found by the zookeeper-master DST world of CAMEL-26974 (PR #30137, branch 
> {{camel-zookeeper-master-dst-world}}), in two cases its default set leaves 
> out: with {{-Ddst.zookeeper.stopWhileSuspended=true}} 7 of 5000 seeds fail, 
> and with {{-Ddst.zookeeper.stopDuringCreate=true}} 100 of the first 1189 
> seeds fail (the run stops at 100), on "once the faults cleared, the master 
> runs its consumer" and "no znode of this node is left in the group after the 
> final stop". Reproduced with a plain unit test (below) against a ZooKeeper 
> server in the JVM, outside the DST harness.
> h3. What happens
> When the master consumer stops or the route of the {{MasterRoutePolicy}} is 
> removed, its {{ZooKeeperGroup}} is closed: it interrupts its thread and 
> deletes its ephemeral znode ({{doUpdate(null)}}). The CuratorFramework is 
> often shared and outlives the group (the client of the component, or the 
> {{curator}} bean), so its session, and every ephemeral znode of the session, 
> lives on. Two cases leave the znode in the group:
> # *Closed while the client is disconnected.* {{doUpdate(null)}} deletes the 
> znode only {{if (isConnected())}}, and then forgets its path ({{id = null}}). 
> When the client reconnects to the same session (a connection loss shorter 
> than the session timeout), the znode is still there.
> # *Closed while the thread of the group waits for the answer of a create.* 
> The interrupt ends the wait, but the create was sent and is applied by the 
> server; the group never records the path, and {{doUpdate(null)}} only logs 
> "Ephemeral node could be created in the registry, but ZooKeeper group didn't 
> record its id".
> The znode stays in the group as long as the session lives, as a member of its 
> own (the container id is new for each join, unless the {{runtime.id}} system 
> property is set). While it has the lowest sequence it is the master of the 
> group, so no node runs the route, this node included after its route started 
> again (it joins behind the left-over znode).
> h3. Expected
> ZooKeeper removes an ephemeral znode only when the session that created it 
> ends (Programmer's Guide, "Ephemeral Nodes"), so a znode a group leaves 
> behind on a live session must be deleted by the client. Curator's 
> {{DeleteBuilder.guaranteed()}} exists for this case: Curator records a failed 
> delete and will "attempt to delete them in the background until successful", 
> as long as the CuratorFramework is open. The znodes of a group carry its uuid 
> ({{NodeState.uuid}}), which {{prunePartialState}} already uses to delete the 
> duplicate of a create whose answer was lost.
> h3. Reproduction
> New {{ZooKeeperGroupCloseTest}} (a {{ZooKeeperServer}} in the JVM, as 
> {{MasterRoutePolicyRouteLifecycleTest}}, and a CuratorFramework with a 30 
> second session):
> * {{groupClosedWhileDisconnectedDeletesItsZnodeOnceReconnected}}: a group 
> joins, the server stops, the group is closed while disconnected, the server 
> starts again on the same data directory and the client reconnects to the same 
> session. On main the znode is still in the group 20 seconds later.
> * {{groupClosedDuringACreateDeletesTheZnode}}: the client applies the create 
> and then waits for the answer until the thread is interrupted (as the 
> ZooKeeper client does when a close interrupts it). On main the znode is still 
> in the group after the close.
> * {{groupClosedDuringACreateBeforeItKnowsItIsConnectedAgainDeletesTheZnode}}: 
> the same, with the group told that the connection was lost and not yet that 
> it is back. On main the znode is still in the group after the close.
> h3. Fix
> In {{ZooKeeperGroup.doUpdate(null)}}:
> * when the client is disconnected, the znode is deleted with 
> {{delete().guaranteed().inBackground()}}: the close does not wait for the 
> connection, and Curator deletes the znode once the client is connected again 
> (if the session expired first, ZooKeeper removed it already). When connected, 
> the foreground guaranteed delete stays as it is.
> * when a create did not answer and the client is connected (the client 
> itself, as the group may not have been told yet that it is connected again: 
> the close removes the listener of the group first), the group reads the 
> children of its path and deletes those with its uuid (the reads come after 
> the create in the order of the session, so they see the znode if the create 
> was applied). While the client is disconnected it still only logs (not 
> covered: the answer of a create lost with the connection, and the close 
> before the reconnect).
> With the fix merged into the world of CAMEL-26974 and two test-only changes 
> to its fake (a guaranteed delete in the background is modelled, and an 
> interrupted retry sleep ends the retries of the client, as Curator's 
> {{SleepingRetry}} does), {{stopDuringCreate}} and {{stopWhileSuspended}} pass 
> 5000 of 5000 seeds each, and the default set 5000 of 5000 (also on main with 
> the changed fake). Branch 
> {{camel-zookeeper-master-group-close-leaves-znode-v2}}.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to