Fixed the problem. It had to do with the leap second bug. Ran the following command on all the three servers and is now working fine.
date;/etc/init.d/ntp stop;/etc/init.d/ntpd stop; date `date +"%m%d%H%M%C%y.%S"`;/etc/init.d/ntp start;/etc/init.d/ntpd start You can find some overview of this problem here: http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/ On Wed, Jul 11, 2012 at 2:35 PM, Narayanan A R < [email protected]> wrote: > Thanks David. Here they are: > > http://pastebin.com/STTnLf9s > http://pastebin.com/PiTgUWpA > http://pastebin.com/4V3AjT34 > > > On Wed, Jul 11, 2012 at 4:32 AM, David Nickerson < > [email protected]> wrote: > >> Narayanan, I don't think the attachments made it through. Can you link to >> the logs in Pastebin? >> >> On Tue, Jul 10, 2012 at 5:24 PM, Narayanan A R < >> [email protected]> wrote: >> >> > Yeah I tried that. Right now I have that set to 2 mins. >> > >> > On Tue, Jul 10, 2012 at 1:47 PM, Jordan Zimmerman < >> > [email protected]> wrote: >> > >> > > Another thing you might need to do is to increase initLimit and >> > syncLimit. >> > > It might be timing out when its syncing. >> > > >> > > -JZ >> > > >> > > On Jul 10, 2012, at 1:46 PM, Narayanan A R wrote: >> > > >> > > > I have 3 servers in the cluster, all bare metal machines and >> installed >> > > ZK 3.4.3. Following is the config in all the servers. >> > > > >> > > > tickTime=2000 >> > > > dataDir=/opt/zookeeper-3.4.3/data >> > > > clientPort=2181 >> > > > initLimit=60 >> > > > syncLimit=60 >> > > > server.1=10.7.78.77:2888:3888 >> > > > server.2=10.7.66.54:2888:3888 >> > > > server.3=10.7.56.61:2888:3888 >> > > > >> > > > When I startup the instances, I see socket timed out in all the >> > > instances. I have attached logs of all the three machine. >> > > > >> > > > Regards, >> > > > ARN >> > > >> > > >> > >> > >
