> > What events are missed for persistent recursive watchers that normal > watcher see?
NodeDeleted, NodeDataChanged and NodeCreated On Fri, Jul 25, 2025 at 1:58 PM Keith Turner <ktur...@apache.org> wrote: > > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote: > > Hi, > > > > I took a look at the code (which I haven't looked at in 5 or more > years). It looks like the reconnection behavior _is_ different. Persistent > watches will miss some events that other watches are getting. This is > indeed a very long-standing bug. > > What events are missed for persistent recursive watchers that normal > watcher see? > > > > > I'd be willing to work on this, but there's likely devs who are more > familiar with the code now who can do it. > > > > -JZ > > > > > On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman < > jor...@jordanzimmerman.com> wrote: > > > > > > Hi, > > > > > > Persistent watches are the same watch as every other watch. It all > goes through the same code. Let's look at the doc: > > > > > > > Because standard watches are one time triggers and there is latency > between getting the event and sending a new request to > > > > get a watch you cannot reliably see every change that happens to a > node in ZooKeeper. Be prepared to handle the case where > > > > the znode changes multiple times between getting the event and > setting the watch again. (You may not care, but at least realize it may > happen.) > > > > > > ZooKeeper does not keep any kind of queue of events. You cannot count > on seeing every event in ZooKeeper. Watchers are triggered as events happen. > > > Again, it's been a very long time since I've looked at the code but > this is my memory of how it works. When I wrote Persistent watches, I used > all > > > the existing watch code. A Persistent watch is the exact same code > path as all other watches. They only difference is that they don't get > deleted after > > > firing. Also, recursive watches trigger for child nodes being watched. > But, again, same code path. > > > > > > I hope this helps. > > > > > > -JZ > > > > > > > > >> On Jul 25, 2025, at 7:30 PM, Li Wang <li4w...@gmail.com> wrote: > > >> > > >> Thanks for the input, Jordan. > > >> > > >> My understanding is that the standard watches do but persistent > watches > > >> don't. Not sure if I miss anything or if this is a bug. Looking > forward to > > >> any feedback/input on this. > > >> > > >> 1. We have the following in the standard watch section of Zookeeper > > >> documentation and it looks like missing notifications are triggered. > > >> > > >> When a client reconnects, any previously registered watches will be > > >>> reregistered and triggered if needed. > > >> > > >> > > >> > > >> > https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics > > >> > > >> > > >> 2. In the code base, Zookeeper client library maintains lastZXid in > memory > > >> and sends it to the server when resetting watches upon reconnection. > The > > >> server detects if any missing notifications need to be triggered > based on > > >> the lastZxid. > > >> > > >> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041 > > >> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497 > > >> > > >> 3. The problem is that missing notifications seem only being > triggered for > > >> standard watches but not for persistent watches when reconnecting. > > >> > > >> For example, for standard watches, watches.process() is invoked for > sending > > >> missing notifications. > > >> > > >> for (String path : dataWatches) { > > >>> DataNode node = getNode(path); > > >>> if (node == null) { > > >>> watcher.process(new > WatchedEvent(EventType.NodeDeleted, > > >>> KeeperState.SyncConnected, path)); > > >>> } else if (node.stat.getMzxid() > relativeZxid) { > > >>> watcher.process(new > > >>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, > path)); > > >>> } else { > > >>> this.dataWatches.addWatch(path, watcher); > > >>> } > > >>> } > > >> > > >> > > >> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521 > > >> > > >> However, for persistence watches, we only register the watches, not > > >> detecting and sending missing notifications. > > >> > > >> for (String path : persistentRecursiveWatches) { > > >>> this.dataWatches.addWatch(path, watcher, > > >>> WatcherMode.PERSISTENT_RECURSIVE); > > >>> } > > >> > > >> > > >> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521 > > >> > > >> Thanks, > > >> > > >> Li > > > > > > > >
