This turned out be a systemctl issue. That was not active for zookeeper. Once that was setup correctly , the observer came up without no problems. It seems like once zookeeper starts up it issues srvr command from localhost . In this case it wasn’t able to open up the socket connections .
> From: [email protected] > To: [email protected] > Subject: Observer not able to connect to the Zk ensemble > Date: Fri, 17 Jul 2015 04:59:26 +0000 > > I am trying to add 2 observers to a zk cluster of 3 nodes > Here is the config > # The number of milliseconds of each tick > tickTime=2000 > # The number of ticks that the initial > # synchronization phase can take > initLimit=10 > # The number client connections per ZK > maxClientCnxns=2000 > # sending a request and getting an acknowledgement > syncLimit=5 > # peerType Participant or Observer > peerType=participant > # the directory where the snapshot is stored. > dataDir=/opt/xxx/data/zookeeper > # the port at which the clients will connect > clientPort=2181 > server.1=xxx:2888:3888:participant > server.2=xxx:2888:3888:participant > server.3=xxx:2888:3888:participant > server.4=xxx:2888:3888:observer > server.5=xxx:2888:3888:observer > > When i connect an observer to Zk ensemble i keep getting this error in the > Leader logs > > 2015-07-16 19:10:00,430 [myid:3] - INFO > [xxx/xxx:3888:QuorumCnxManager$Listener@511] - Received connection request > /xxx:47729 > 2015-07-16 19:10:00,431 [myid:3] - DEBUG > [xxx/xxx:3888:QuorumCnxManager$SendWorker@588] - Address of remote peer: 4 > 2015-07-16 19:10:00,433 [myid:3] - DEBUG > [WorkerSender[myid=3]:QuorumCnxManager@387] - There is a connection already > for server 4 > 2015-07-16 19:10:01,059 [myid:3] - WARN > [RecvWorker:4:QuorumCnxManager$RecvWorker@780] - Connection broken for id 4, > my id = 3, error = > java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:392) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765) > 2015-07-16 19:10:01,060 [myid:3] - WARN > [RecvWorker:4:QuorumCnxManager$RecvWorker@783] - Interrupting SendWorker > 2015-07-16 19:10:01,060 [myid:3] - DEBUG > [RecvWorker:4:QuorumCnxManager$SendWorker@606] - Calling finish for 4 > 2015-07-16 19:10:01,060 [myid:3] - DEBUG > [RecvWorker:4:QuorumCnxManager$SendWorker@626] - Removing entry from > senderWorkerMap sid=4 > 2015-07-16 19:10:01,060 [myid:3] - WARN > [SendWorker:4:QuorumCnxManager$SendWorker@697] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:849) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:64) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:685) > 2015-07-16 19:10:01,061 [myid:3] - DEBUG > [SendWorker:4:QuorumCnxManager$SendWorker@606] - Calling finish for 4 > 2015-07-16 19:10:01,061 [myid:3] - WARN > [SendWorker:4:QuorumCnxManager$SendWorker@706] - Send worker leaving thread > > Ther other observer no 5 connects without any issues > > Any ideas ? >
