Boyi Dai created ZOOKEEPER-5129:
-----------------------------------

             Summary: Container cleanup can delete a node that was created at 
the same path after the container was picked for deletion
                 Key: ZOOKEEPER-5129
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5129
             Project: ZooKeeper
          Issue Type: Bug
          Components: server
    Affects Versions: 3.9.7
            Reporter: Boyi Dai


{{ContainerManager.checkContainers}} first collects the paths of the containers 
(and expired TTL nodes) to delete, then sends a {{DeleteContainerRequest}} for 
each path in turn, waiting between requests to stay under 
{{znode.container.maxPerMinute}}. The request only carries the path.

If a candidate is deleted and recreated by a client before its request is 
processed, {{PrepRequestProcessor}} deletes the node that is now at that path. 
Its {{deleteContainer}} case only rejects a node that has children or is a 
normal ephemeral node. A persistent node has {{ephemeralOwner}} 0, which 
{{EphemeralType.get}} returns as {{VOID}}, so it passes. A new container also 
passes, even though a container that was never used is meant to be kept for 
{{znode.container.maxNeverUsedInterval}}.

So a persistent node with data, created successfully by a client, can be 
deleted by container cleanup, and a new container or TTL node can be deleted 
right after it is created. A TTL node that is modified after it was picked, 
which restarts its TTL, is deleted as well.

This was found through code analysis, and I reproduced each case with tests 
that replace or modify the node between {{ContainerManager}} picking it and its 
delete request being processed.

*Steps to reproduce:*
# Create container {{/foo}}, add a child and delete it, so {{/foo}} is empty 
and becomes a cleanup candidate.
# After {{ContainerManager}} picks {{/foo}} but before its delete request is 
processed, delete {{/foo}} and create a persistent node {{/foo}} with data.
# The delete request deletes the persistent {{/foo}}.

*Expected:* container cleanup only deletes the container or TTL node it picked, 
and keeps a node that replaced it.

*Proposed fix:*
* In {{ContainerManager}}, check each candidate again right before sending its 
request, skip it if it no longer exists or is no longer past its grace period, 
and send the node's {{mzxid}} with the request.
* In {{PrepRequestProcessor}}, only delete a container or TTL node, and only if 
its {{mzxid}} is unchanged, so a node that is replaced or modified after that 
check is kept, even if the change isn't committed yet.
* {{DeleteContainerRequest}} is only created by {{ContainerManager}} on the 
leader, so adding the {{mzxid}} doesn't change anything sent between servers or 
written to the log.

I have a patch with tests and will open a PR.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to