> > All we have with a persistent recursive watcher is the root ZNode that is > being watched. The only thing we can determine on reconnect is that the > zxid of child nodes are later than the root node's zxid. There is no way to > know if nodes have been deleted. We can't even know if a node is new or > changed right? I don't see how this can work.
Yes, for a persistent recursive watch, that's the challenge I saw too. Not sure if someone has a good solution for it. How about having an option to send notification indicating the subtree has been changed, even not able to tell exactly what are the changes. This can help the client app to determine whether it needs to reset the application state. For persistent non-recursive watches, we can tell exactly the changes just like the standard watches, right? On Mon, Jul 28, 2025 at 1:03 PM Jordan Zimmerman <jor...@jordanzimmerman.com> wrote: > Actually, thinking about this more in the recursive watcher case... is > this even possible? All we have with a persistent recursive watcher is the > root ZNode that is being watched. The only thing we can determine on > reconnect is that the zxid of child nodes are later than the root node's > zxid. There is no way to know if nodes have been deleted. We can't even > know if a node is new or changed right? I don't see how this can work. > Maybe someone else does. > > -JZ > > > On Jul 28, 2025, at 8:40 PM, Jordan Zimmerman < > jor...@jordanzimmerman.com> wrote: > > > > To be clear, we can easily get these for the root/base node of the > Persistent watcher (just like other watchers). For recursive watchers, > however, we'd have to walk the entire tree and infer changes from the zxid. > It could result in watchers firing for every node in the tree on reconnect. > > > > -JZ > > > >> On Jul 28, 2025, at 8:11 PM, Li Wang <li4w...@gmail.com> wrote: > >> > >>> > >>> What events are missed for persistent recursive watchers that normal > >>> watcher see? > >> > >> > >> NodeDeleted, NodeDataChanged and NodeCreated > >> > >> On Fri, Jul 25, 2025 at 1:58 PM Keith Turner <ktur...@apache.org> > wrote: > >> > >>> > >>> > >>> On 2025/07/25 19:23:41 Jordan Zimmerman wrote: > >>>> Hi, > >>>> > >>>> I took a look at the code (which I haven't looked at in 5 or more > >>> years). It looks like the reconnection behavior _is_ different. > Persistent > >>> watches will miss some events that other watches are getting. This is > >>> indeed a very long-standing bug. > >>> > >>> What events are missed for persistent recursive watchers that normal > >>> watcher see? > >>> > >>>> > >>>> I'd be willing to work on this, but there's likely devs who are more > >>> familiar with the code now who can do it. > >>>> > >>>> -JZ > >>>> > >>>>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman < > >>> jor...@jordanzimmerman.com> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> Persistent watches are the same watch as every other watch. It all > >>> goes through the same code. Let's look at the doc: > >>>>> > >>>>>> Because standard watches are one time triggers and there is latency > >>> between getting the event and sending a new request to > >>>>>> get a watch you cannot reliably see every change that happens to a > >>> node in ZooKeeper. Be prepared to handle the case where > >>>>>> the znode changes multiple times between getting the event and > >>> setting the watch again. (You may not care, but at least realize it may > >>> happen.) > >>>>> > >>>>> ZooKeeper does not keep any kind of queue of events. You cannot count > >>> on seeing every event in ZooKeeper. Watchers are triggered as events > happen. > >>>>> Again, it's been a very long time since I've looked at the code but > >>> this is my memory of how it works. When I wrote Persistent watches, I > used > >>> all > >>>>> the existing watch code. A Persistent watch is the exact same code > >>> path as all other watches. They only difference is that they don't get > >>> deleted after > >>>>> firing. Also, recursive watches trigger for child nodes being > watched. > >>> But, again, same code path. > >>>>> > >>>>> I hope this helps. > >>>>> > >>>>> -JZ > >>>>> > >>>>> > >>>>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li4w...@gmail.com> wrote: > >>>>>> > >>>>>> Thanks for the input, Jordan. > >>>>>> > >>>>>> My understanding is that the standard watches do but persistent > >>> watches > >>>>>> don't. Not sure if I miss anything or if this is a bug. Looking > >>> forward to > >>>>>> any feedback/input on this. > >>>>>> > >>>>>> 1. We have the following in the standard watch section of Zookeeper > >>>>>> documentation and it looks like missing notifications are triggered. > >>>>>> > >>>>>> When a client reconnects, any previously registered watches will be > >>>>>>> reregistered and triggered if needed. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>> > https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics > >>>>>> > >>>>>> > >>>>>> 2. In the code base, Zookeeper client library maintains lastZXid in > >>> memory > >>>>>> and sends it to the server when resetting watches upon reconnection. > >>> The > >>>>>> server detects if any missing notifications need to be triggered > >>> based on > >>>>>> the lastZxid. > >>>>>> > >>>>>> > >>> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041 > >>>>>> > >>> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497 > >>>>>> > >>>>>> 3. The problem is that missing notifications seem only being > >>> triggered for > >>>>>> standard watches but not for persistent watches when reconnecting. > >>>>>> > >>>>>> For example, for standard watches, watches.process() is invoked for > >>> sending > >>>>>> missing notifications. > >>>>>> > >>>>>> for (String path : dataWatches) { > >>>>>>> DataNode node = getNode(path); > >>>>>>> if (node == null) { > >>>>>>> watcher.process(new > >>> WatchedEvent(EventType.NodeDeleted, > >>>>>>> KeeperState.SyncConnected, path)); > >>>>>>> } else if (node.stat.getMzxid() > relativeZxid) { > >>>>>>> watcher.process(new > >>>>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, > >>> path)); > >>>>>>> } else { > >>>>>>> this.dataWatches.addWatch(path, watcher); > >>>>>>> } > >>>>>>> } > >>>>>> > >>>>>> > >>>>>> > >>> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521 > >>>>>> > >>>>>> However, for persistence watches, we only register the watches, not > >>>>>> detecting and sending missing notifications. > >>>>>> > >>>>>> for (String path : persistentRecursiveWatches) { > >>>>>>> this.dataWatches.addWatch(path, watcher, > >>>>>>> WatcherMode.PERSISTENT_RECURSIVE); > >>>>>>> } > >>>>>> > >>>>>> > >>>>>> > >>> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521 > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Li > >>>>> > >>>> > >>>> > >>> > > > >
