Hi, We have a simple architecture: 2 SOLR Cloud servers (on servers #1 and #2), and 3 zookeeper instances (on servers #1, #2, and #3). Things appear to work fine but:
We are getting *TONS* of continuous log warnings from our client applications. From one server it shows this: [MYAPP-WEB] 2019-01-03 14:17:46,519 WARN [org.apache.solr.common.cloud.ConnectionManager] - <Watcher org.apache.solr.common.cloud.ConnectionManager@a38d23a name: ZooKeeperConnection Watcher:888.88.101.65:2181,888.88.101.64:2181,888.88.101.63:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None> [MYAPP-WEB] 2019-01-03 14:17:46,519 WARN [org.apache.solr.common.cloud.ConnectionManager] - <zkClient has disconnected> [MYAPP-WEB] 2019-01-03 14:17:47,385 INFO [org.apache.zookeeper.ClientCnxn] - <Opening socket connection to server 888.88.101.64/888.88.101.64:2181. Will not attempt to authenticate using SASL (unknown error)> [MYAPP-WEB] 2019-01-03 14:17:47,386 INFO [org.apache.zookeeper.ClientCnxn] - <Socket connection established to 888.88.101.64/888.88.101.64:2181, initiating session> [MYAPP-WEB] 2019-01-03 14:17:47,386 INFO [org.apache.zookeeper.ClientCnxn] - <Session establishment complete on server 888.88.101.64/888.88.101.64:2181, sessionid = 0x2679a04fc7b0004, negotiated timeout = 10000> [MYAPP-WEB] 2019-01-03 14:17:47,386 INFO [org.apache.solr.common.cloud.ConnectionManager] - <zkClient has connected> [MYAPP-WEB] 2019-01-03 14:17:47,386 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x2679a04fc7b0004 for server 888.88.101.64/888.88.101.64:2181, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/WatcherEvent at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:770) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) [MYAPP-WEB] 2019-01-03 14:17:47,487 WARN [org.apache.solr.common.cloud.ConnectionManager] - <Watcher org.apache.solr.common.cloud.ConnectionManager@a38d23a name: ZooKeeperConnection Watcher:888.88.101.65:2181,888.88.101.64:2181,888.88.101.63:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None> [MYAPP-WEB] 2019-01-03 14:17:47,487 WARN [org.apache.solr.common.cloud.ConnectionManager] - <zkClient has disconnected> [MYAPP-WEB] 2019-01-03 14:17:47,943 INFO [org.apache.zookeeper.ClientCnxn] - <Opening socket connection to server 888.88.101.63/888.88.101.63:2181. Will not attempt to authenticate using SASL (unknown error)> [MYAPP-WEB] 2019-01-03 14:17:47,943 INFO [org.apache.zookeeper.ClientCnxn] - <Socket connection established to 888.88.101.63/888.88.101.63:2181, initiating session> [MYAPP-WEB] 2019-01-03 14:17:47,943 INFO [org.apache.zookeeper.ClientCnxn] - <Session establishment complete on server 888.88.101.63/888.88.101.63:2181, sessionid = 0x2679a04fc7b0004, negotiated timeout = 10000> [MYAPP-WEB] 2019-01-03 14:17:47,944 INFO [org.apache.solr.common.cloud.ConnectionManager] - <zkClient has connected> [MYAPP-WEB] 2019-01-03 14:17:47,944 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x2679a04fc7b0004 for server 888.88.101.63/888.88.101.63:2181, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/WatcherEvent at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:770) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) [MYAPP-WEB] 2019-01-03 14:17:48,044 WARN [org.apache.solr.common.cloud.ConnectionManager] - <Watcher org.apache.solr.common.cloud.ConnectionManager@a38d23a name: ZooKeeperConnection Watcher:888.88.101.65:2181,888.88.101.64:2181,888.88.101.63:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None> [MYAPP-WEB] 2019-01-03 14:17:48,044 WARN [org.apache.solr.common.cloud.ConnectionManager] - <zkClient has disconnected> [MYAPP-WEB] 2019-01-03 14:17:48,687 INFO [org.apache.zookeeper.ClientCnxn] - <Opening socket connection to server 888.88.101.65/888.88.101.65:2181. Will not attempt to authenticate using SASL (unknown error)> [MYAPP-WEB] 2019-01-03 14:17:48,687 INFO [org.apache.zookeeper.ClientCnxn] - <Socket connection established to 888.88.101.65/888.88.101.65:2181, initiating session> [MYAPP-WEB] 2019-01-03 14:17:48,688 INFO [org.apache.zookeeper.ClientCnxn] - <Session establishment complete on server 888.88.101.65/888.88.101.65:2181, sessionid = 0x2679a04fc7b0004, negotiated timeout = 10000> [MYAPP-WEB] 2019-01-03 14:17:48,689 INFO [org.apache.solr.common.cloud.ConnectionManager] - <zkClient has connected> [MYAPP-WEB] 2019-01-03 14:17:48,689 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x2679a04fc7b0004 for server 888.88.101.65/888.88.101.65:2181, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/WatcherEvent at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:770) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) And from another server we get this: [MYAPP-WEB] 2019-01-03 14:19:47,273 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x3679a09efbb0002 for server null, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/Login at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:216) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:119) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1011) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) [MYAPP-WEB] 2019-01-03 14:19:47,753 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x3679a09efbb0002 for server null, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/Login at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:216) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:119) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1011) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) [MYAPP-WEB] 2019-01-03 14:19:48,197 INFO [gov.fbi.guardian.web.filter.SentinelRedirectFilter] - <Sentinel Redirect touched> [MYAPP-WEB] 2019-01-03 14:19:48,450 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x3679a09efbb0002 for server null, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/Login at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:216) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:119) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1011) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) [MYAPP-WEB] 2019-01-03 14:19:49,912 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x3679a09efbb0002 for server null, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/Login at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:216) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:119) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1011) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) [MYAPP-WEB] 2019-01-03 14:19:50,977 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x3679a09efbb0002 for server null, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/Login at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:216) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:119) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1011) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) [MYAPP-WEB] 2019-01-03 14:19:51,233 WARN [org.apache.zookeeper.ClientCnxn] - <Session 0x3679a09efbb0002 for server null, unexpected error, closing socket connection and attempting reconnect> java.lang.NoClassDefFoundError: org/apache/zookeeper/Login at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:216) at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:119) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1011) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) These are making our application logs impossible to read, and I assume indicate that something is wrong. Thanks for any help! Joe Lerner -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html