> > So we don't know whether an absent node deserves `NodeDeleted` or not > from `DataTree`.
That's right. We have an issue even with persistent watches. Persistent watch is added via addWatch() API and it can be attached to a non-existent node. When we detect a node doesn’t exist upon reconnecting, we can not tell whether the node never exists or is being deleted. On Mon, Jul 28, 2025 at 8:29 PM Kezhu Wang <kez...@gmail.com> wrote: > Hi, > > > To be clear, we can easily get these for the root/base node of the > Persistent watcher (just like other watchers). > > > For persistent non-recursive watches, we can tell exactly the changes > just > like the standard watches, right? > > Persistent watch is resistant to node deletion[1], this means we can > attach it to no exist nodes and it won't be deleted after node > deletion. So we don't know whether an absent node deserves > `NodeDeleted` or not from `DataTree`. > > > What happens if the rebuilding cache needs to read millions of znodes? > > You probably should avoid running into such a situation. Or you could > use standard watches, and bookkeep every notification until the > reaction succeeds. > > [1]: > https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/main/java/org/apache/zookeeper/AddWatchMode.java#L25-L34 > > Best, > Kezhu Wang > > On Tue, Jul 29, 2025 at 9:40 AM Li Wang <li4w...@gmail.com> wrote: > > > > > > > > In the case of building a cache, I recommend rebuilding the cache after > > > reconnection > > > > > > What happens if the rebuilding cache needs to read millions of zndoes? > > > > On Sun, Jul 27, 2025 at 1:37 AM Kezhu Wang <kez...@gmail.com> wrote: > > > > > Hi all, > > > > > > I think Jordan is correct about this. > > > > > > > In the case of reconnection, is it at least guaranteed you'll get > some > > > kind of client notification about the connection state, so you can > > > reset any application state that relies on watchers being notified? > > > Like, you may not see a node changed during the disconnected state, > > > but will you at least get a connected notification from the persistent > > > watcher? Or do you have to rely on the connection status watcher set > > > when the client was created to see those? > > > > > > It is guaranteed that persistent watchers will get state notifications > > > in the same channel(or callback) as node change events. There is a > > > test for this.[1] > > > > > > > The problem is that missing notifications seem only being triggered > for > > > standard watches but not for persistent watches when reconnecting. > > > > > > This is misleading. What is sent to the client are not "missing > > > notifications", but simply the last state, so all intermediate changes > > > are lost. Jardan has pointed out this. This is what the doc states: > > > > > > > There is one case where a watch may be missed: a watch for the > existence > > > of a znode not yet created will be missed if the znode is created and > > > deleted while disconnected. > > > > > > Basically, we are firing node change events based on `DataTree`(a.k.a. > > > snapshot) but not log entries. > > > > > > In the case of building a cache, I recommend rebuilding the cache > > > after reconnection, it is the safest option from my point of view. [2] > > > This is also Apache Curator's handling of disconnection. > > > > > > [1]: > > > > https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/test/java/org/apache/zookeeper/test/PersistentRecursiveWatcherTest.java#L151-L161 > > > [2]: > https://github.com/apache/zookeeper/pull/1950#issuecomment-1557685392 > > > > > > Best, > > > Kezhu Wang > > > > > > > > > > > > On Sat, Jul 26, 2025 at 10:26 AM Kezhu Wang <kez...@gmail.com> wrote: > > > > > > > > Hi all, > > > > > > > > There is a jira issue: > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links > to > > > > more context. > > > > > > > > Best, > > > > Kezhu Wang > > > > > > > > On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman > > > > <jor...@jordanzimmerman.com> wrote: > > > > > > > > > > Here's a summary: > > > > > > > > > > On reconnect, watches are reset. For Data watches, if the node no > > > longer exists, the watch will get NodeDeleted. If the node's zxId is > > > different, the watch will get NodeDataChanged. Exist and child nodes > have > > > similar handling. Persistent watches, on the other hand, are merely > reset. > > > > > > > > > > I no longer remember why we didn't mimic this for Persistent > watches. > > > I guess it can be argued that it isn't necessary or that it could > result if > > > a _lot_ of persistent watch calls. Maybe the right thing to do is to > just > > > document the difference and leave it as it's been this way for years. > > > > > > > > > > -Jordan > > > > > > > > > > > On Jul 25, 2025, at 9:58 PM, Keith Turner <ktur...@apache.org> > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote: > > > > > >> Hi, > > > > > >> > > > > > >> I took a look at the code (which I haven't looked at in 5 or > more > > > years). It looks like the reconnection behavior _is_ different. > Persistent > > > watches will miss some events that other watches are getting. This is > > > indeed a very long-standing bug. > > > > > > > > > > > > What events are missed for persistent recursive watchers that > normal > > > watcher see? > > > > > > > > > > > >> > > > > > >> I'd be willing to work on this, but there's likely devs who are > > > more familiar with the code now who can do it. > > > > > >> > > > > > >> -JZ > > > > > >> > > > > > >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman < > > > jor...@jordanzimmerman.com> wrote: > > > > > >>> > > > > > >>> Hi, > > > > > >>> > > > > > >>> Persistent watches are the same watch as every other watch. It > all > > > goes through the same code. Let's look at the doc: > > > > > >>> > > > > > >>>> Because standard watches are one time triggers and there is > > > latency between getting the event and sending a new request to > > > > > >>>> get a watch you cannot reliably see every change that happens > to > > > a node in ZooKeeper. Be prepared to handle the case where > > > > > >>>> the znode changes multiple times between getting the event and > > > setting the watch again. (You may not care, but at least realize it may > > > happen.) > > > > > >>> > > > > > >>> ZooKeeper does not keep any kind of queue of events. You cannot > > > count on seeing every event in ZooKeeper. Watchers are triggered as > events > > > happen. > > > > > >>> Again, it's been a very long time since I've looked at the code > > > but this is my memory of how it works. When I wrote Persistent > watches, I > > > used all > > > > > >>> the existing watch code. A Persistent watch is the exact same > code > > > path as all other watches. They only difference is that they don't get > > > deleted after > > > > > >>> firing. Also, recursive watches trigger for child nodes being > > > watched. But, again, same code path. > > > > > >>> > > > > > >>> I hope this helps. > > > > > >>> > > > > > >>> -JZ > > > > > >>> > > > > > >>> > > > > > >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li4w...@gmail.com> > wrote: > > > > > >>>> > > > > > >>>> Thanks for the input, Jordan. > > > > > >>>> > > > > > >>>> My understanding is that the standard watches do but > persistent > > > watches > > > > > >>>> don't. Not sure if I miss anything or if this is a bug. > Looking > > > forward to > > > > > >>>> any feedback/input on this. > > > > > >>>> > > > > > >>>> 1. We have the following in the standard watch section of > > > Zookeeper > > > > > >>>> documentation and it looks like missing notifications are > > > triggered. > > > > > >>>> > > > > > >>>> When a client reconnects, any previously registered watches > will > > > be > > > > > >>>>> reregistered and triggered if needed. > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics > > > > > >>>> > > > > > >>>> > > > > > >>>> 2. In the code base, Zookeeper client library maintains > lastZXid > > > in memory > > > > > >>>> and sends it to the server when resetting watches upon > > > reconnection. The > > > > > >>>> server detects if any missing notifications need to be > triggered > > > based on > > > > > >>>> the lastZxid. > > > > > >>>> > > > > > >>>> > > > > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041 > > > > > >>>> > > > > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497 > > > > > >>>> > > > > > >>>> 3. The problem is that missing notifications seem only being > > > triggered for > > > > > >>>> standard watches but not for persistent watches when > reconnecting. > > > > > >>>> > > > > > >>>> For example, for standard watches, watches.process() is > invoked > > > for sending > > > > > >>>> missing notifications. > > > > > >>>> > > > > > >>>> for (String path : dataWatches) { > > > > > >>>>> DataNode node = getNode(path); > > > > > >>>>> if (node == null) { > > > > > >>>>> watcher.process(new > > > WatchedEvent(EventType.NodeDeleted, > > > > > >>>>> KeeperState.SyncConnected, path)); > > > > > >>>>> } else if (node.stat.getMzxid() > relativeZxid) { > > > > > >>>>> watcher.process(new > > > > > >>>>> WatchedEvent(EventType.NodeDataChanged, > > > KeeperState.SyncConnected, path)); > > > > > >>>>> } else { > > > > > >>>>> this.dataWatches.addWatch(path, watcher); > > > > > >>>>> } > > > > > >>>>> } > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521 > > > > > >>>> > > > > > >>>> However, for persistence watches, we only register the > watches, > > > not > > > > > >>>> detecting and sending missing notifications. > > > > > >>>> > > > > > >>>> for (String path : persistentRecursiveWatches) { > > > > > >>>>> this.dataWatches.addWatch(path, watcher, > > > > > >>>>> WatcherMode.PERSISTENT_RECURSIVE); > > > > > >>>>> } > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521 > > > > > >>>> > > > > > >>>> Thanks, > > > > > >>>> > > > > > >>>> Li > > > > > >>> > > > > > >> > > > > > >> > > > > > > > > >
