Boyi Dai created ZOOKEEPER-5129:
-----------------------------------
Summary: Container cleanup can delete a node that was created at
the same path after the container was picked for deletion
Key: ZOOKEEPER-5129
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5129
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.9.7
Reporter: Boyi Dai
{{ContainerManager.checkContainers}} first collects the paths of the containers
(and expired TTL nodes) to delete, then sends a {{DeleteContainerRequest}} for
each path in turn, waiting between requests to stay under
{{znode.container.maxPerMinute}}. The request only carries the path.
If a candidate is deleted and recreated by a client before its request is
processed, {{PrepRequestProcessor}} deletes the node that is now at that path.
Its {{deleteContainer}} case only rejects a node that has children or is a
normal ephemeral node. A persistent node has {{ephemeralOwner}} 0, which
{{EphemeralType.get}} returns as {{VOID}}, so it passes. A new container also
passes, even though a container that was never used is meant to be kept for
{{znode.container.maxNeverUsedInterval}}.
So a persistent node with data, created successfully by a client, can be
deleted by container cleanup, and a new container or TTL node can be deleted
right after it is created. A TTL node that is modified after it was picked,
which restarts its TTL, is deleted as well.
This was found through code analysis, and I reproduced each case with tests
that replace or modify the node between {{ContainerManager}} picking it and its
delete request being processed.
*Steps to reproduce:*
# Create container {{/foo}}, add a child and delete it, so {{/foo}} is empty
and becomes a cleanup candidate.
# After {{ContainerManager}} picks {{/foo}} but before its delete request is
processed, delete {{/foo}} and create a persistent node {{/foo}} with data.
# The delete request deletes the persistent {{/foo}}.
*Expected:* container cleanup only deletes the container or TTL node it picked,
and keeps a node that replaced it.
*Proposed fix:*
* In {{ContainerManager}}, check each candidate again right before sending its
request, skip it if it no longer exists or is no longer past its grace period,
and send the node's {{mzxid}} with the request.
* In {{PrepRequestProcessor}}, only delete a container or TTL node, and only if
its {{mzxid}} is unchanged, so a node that is replaced or modified after that
check is kept, even if the change isn't committed yet.
* {{DeleteContainerRequest}} is only created by {{ContainerManager}} on the
leader, so adding the {{mzxid}} doesn't change anything sent between servers or
written to the log.
I have a patch with tests and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)