All, I have multi data-center ldap cluster setup with other data-center with all observers all of sudden all the observer threads went down with the following message, any idea why they went down? We don't see any network related issues between data-centers.
2018-06-29 05:32:59,036 [myid:222] - WARN [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@79] - Exception when observing the leader java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152) at org.apache.zookeeper.server.quorum.Observer.observeLeader(Observer.java:75) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727) 2018-06-29 05:32:59,244 [myid:222] - INFO [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@137] - shutdown called java.lang.Exception: shutdown Observer at org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:731) Thanks, Ram
