Hi Curator folks,
We hit an issue when implementing rolling upgrade to Apache Oozie
service with Curator framework 5.4.0 and ZooKeeper 3.8.
Original setup:
Oozie with Curator which doesn't use persisent watchers
ZooKeeper server 3.5.5
New setup:
Oozie with Curator 5.4.0
ZooKeeper server 3.8.0
We upgrade Oozie first which means new Curator should run on Zookeeper
server version 3.5.5. Oozie doesn't use CuratorCache, so persistent
watchers should not being used.
Curators compatibility check doesn't work, because it only checks the
ZK client version (3.8.0), so using persistent watchers is enabled
within Curator.
We get the following exception on the client side:
2023-08-21 03:42:54,163 INFO THREAD[main-EventThread]
org.apache.curator.framework.state.ConnectionStateManager:
SERVER[quasar-lxqhyq-1.quasar-lxqhyq.root.hwx.site] State change:
RECONNECTED
2023-08-21 03:42:54,187 ERROR THREAD[epollEventLoopGroup-2-1]
org.apache.zookeeper.ClientCnxnSocketNetty: SERVER[quasar-lxqhyq-
1.quasar-lxqhyq.root.hwx.site] Unexpected throwable
java.io.IOException: Xid out of order. Got Xid 32 with err -6 expected
Xid 30 for a packet with details: clientPath:null serverPath:null
finished:false header:: 30,101 replyHeader:: 0,0,-4 request::
8589963026,v{'/zookeeper/config},v{},v{} response:: null
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java
:954)
And on the server side:
2023-08-21 03:42:54,673 WARN
org.apache.zookeeper.server.ZooKeeperServer: Received packet at server
of unknown type 106
OpCode.addWatch == 106 which means somebody is trying to do addWatch.
Is CuratorCache used internally in Curator? What could the client try
to do by adding watcher on /zookeeper/config node?
Is the workflow supported or should we always upgrade ZK server first?
Thanks,
Andor
