Boyi Dai created ZOOKEEPER-5117:
-----------------------------------
Summary: setquota can leave a quota half created, which its own
retry can't fix and which isn't enforced
Key: ZOOKEEPER-5117
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5117
Project: ZooKeeper
Issue Type: Bug
Components: quota
Affects Versions: 3.9.7
Reporter: Boyi Dai
SetQuotaCommand.createQuota creates the quota's zookeeper_limits node and then
its zookeeper_stats node with two separate create calls. If the call fails
after the first create (for example a connection loss, or a lost response to
the first create), only the limit node exists.
Retrying the same setquota then takes the "quota already exists" branch, which
reads zookeeper_stats before writing anything and fails with NoNode. Every
retry fails the same way, so the quota can only be fixed by deleting or
recreating its nodes by hand.
Meanwhile the limit node registers the quota with the server, but
ZooKeeperServer.checkQuota and DataTree.updateQuotaStat log and return when the
stat node is missing. With zookeeper.enforceQuota=true, a hard count or byte
limit is then silently not enforced.
The reverse can also happen: delquota deletes the two nodes one at a time, so a
stat node can be left without a limit node. A later setquota then fails with
NodeExists on the stat node. A retry succeeds, but keeps the old stat node,
whose usage stopped being updated when the limit node was removed.
This was found through code analysis, and I reproduced both cases with tests
that create the partial quota nodes directly.
Steps to reproduce:
1. Create /ns with a child.
2. Create /zookeeper/quota/ns/zookeeper_limits, but not zookeeper_stats (the
state left when setquota fails between its two creates).
3. Run setquota on /ns: it fails with NoNode for
/zookeeper/quota/ns/zookeeper_stats, on every retry.
4. With enforceQuota=true and a hard count limit, creates under /ns that exceed
the limit still succeed.
Expected: setquota shouldn't leave a partial quota behind, and running it again
should complete a quota that was left partial.
Proposed fix: when creating a new quota, create the limit and stat nodes
together in one multi (replacing a stat node left without its limit node), and
when the limit node exists but the stat node doesn't, create the stat node. The
server already computes the real usage when a stat node is created. delquota
would also delete the limit and stat nodes in one multi. I have a patch with
tests and will open a PR.
Not covered by this fix: a quota that already has both nodes but a stale stat
node (left by an earlier failed setquota retry) keeps its stale usage; delquota
followed by setquota fixes it. The server still doesn't enforce a quota whose
stat node is missing; the fix makes setquota stop leaving that state and repair
it, rather than changing the server.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
