Hi-
I'm running a five-node Accumulo 1.4.5 cluster with zookeeper 3.4.6
distributed across the same systems.
We've seen a couple tserver failures in a manifestation that feels similar
to ACCUMULO-1572 (which was patched in 1.4.5). What is perhaps unique in
this circumstance is that the user reported these failures occurring
immediately upon entering a command in the accumulo shell. The commands
were a routine scan and delete. The error is attached but boils down to:
2014-04-09 21:48:49,552 [zookeeper.ZooLock] WARN : lost connection to
> zookeeper
> 2014-04-09 21:48:49,552 [zookeeper.ZooCache] WARN : Zookeeper error, will
> retry
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /acc....
> 2014-04-09 21:48:49,554 [zookeeper.DistributedWorkQueue] INFO : Got
> unexpected zookeeper event: None
> [ repeat the above a few times and then finally ]
> 2014-04-09 21:48:51,866 [tabletserver.TabletServer] FATAL: Lost ability to
> monitor tablet server lock, exiting.
The zookeeper arrangement here is non-optimal in that they're working on
the same virtualized disk as the hadoop and accumulo processes. The system
was performing bulk ingest at the time so contention was very likely an
issue.
Zookeeper did report, at essentially the same millisecond:
2014-04-09 21:48:49,551 [myid:1] - WARN
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when
> following the leader
> java.net.SocketTimeoutException: Read timed out
> [ followed by a number of ]
> 2014-04-09 21:48:49,919 [myid:1] - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of
> session 0x0 due to java.io.IOException: ZooKeeperServer not running
It's important to note however that:
- The ZooKeeper errors above occur many other times in the logs and the
accumulo cluster has been ok.
- The ZooKeeper ensemble recovered without intervention.
- The WARN to FATAL time for Accumulo was just two seconds whereas I was
under the impression the process would only give up after two retry
attempts lasting 30s each.
- Only the tserver on the system where the user was running the accumulo
shell failed and only (we believe) upon issuance of a command.
- accumulo-site.xml on all nodes is configured with three zookeepers so the
system should be attempting to fail over.
Thanks,
Frans
2014-04-09 21:48:45,961 [server.Accumulo] WARN : System swappiness setting is
greater than ten (60) which can cause time-sensitive operations to be delayed.
Accumulo is time sensitive because it needs to maintain distributed lock
agreement.
2014-04-09 21:48:49,552 [zookeeper.ZooLock] WARN : lost connection to zookeeper
2014-04-09 21:48:49,552 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/users/redact
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.security.ZKAuthenticator.authenticate(ZKAuthenticator.java:97)
at
org.apache.accumulo.server.security.ZKAuthenticator.authenticateUser(ZKAuthenticator.java:184)
at
org.apache.accumulo.server.security.Auditor.authenticateUser(Auditor.java:69)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startUpdate(TabletServer.java:1366)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at com.sun.proxy.$Proxy2.startUpdate(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startUpdate.process(TabletClientService.java:2294)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2053)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:49,554 [zookeeper.DistributedWorkQueue] INFO : Got unexpected
zookeeper event: None for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/bulk_failed_copyq
2014-04-09 21:48:49,555 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/config/table.compaction.minor.idle
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.conf.ZooConfiguration.get(ZooConfiguration.java:109)
at
org.apache.accumulo.server.conf.ZooConfiguration._get(ZooConfiguration.java:76)
at
org.apache.accumulo.server.conf.ZooConfiguration.get(ZooConfiguration.java:103)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:114)
at
org.apache.accumulo.core.conf.AccumuloConfiguration.getTimeInMillis(AccumuloConfiguration.java:83)
at
org.apache.accumulo.server.tabletserver.LargestFirstMemoryManager.getMemoryManagementActions(LargestFirstMemoryManager.java:95)
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager$MemoryManagementFramework.manageMemory(TabletServerResourceManager.java:317)
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager$MemoryManagementFramework.access$200(TabletServerResourceManager.java:229)
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager$MemoryManagementFramework$2.run(TabletServerResourceManager.java:254)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,198 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/config/table.bloom.enabled
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.conf.ZooConfiguration.get(ZooConfiguration.java:109)
at
org.apache.accumulo.server.conf.ZooConfiguration._get(ZooConfiguration.java:76)
at
org.apache.accumulo.server.conf.ZooConfiguration.get(ZooConfiguration.java:103)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:114)
at
org.apache.accumulo.core.conf.AccumuloConfiguration.getBoolean(AccumuloConfiguration.java:108)
at
org.apache.accumulo.core.file.DispatchingFileFactory.openReader(FileOperations.java:72)
at
org.apache.accumulo.server.tabletserver.Compactor.call(Compactor.java:150)
at
org.apache.accumulo.server.tabletserver.MinorCompactor.call(MinorCompactor.java:80)
at
org.apache.accumulo.server.tabletserver.Tablet.minorCompact(Tablet.java:2162)
at
org.apache.accumulo.server.tabletserver.Tablet.access$4400(Tablet.java:145)
at
org.apache.accumulo.server.tabletserver.Tablet$MinorCompactionTask.run(Tablet.java:2241)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,198 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/tables/6/conf/table.bloom.enabled
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:121)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:109)
at
org.apache.accumulo.core.conf.AccumuloConfiguration.getBoolean(AccumuloConfiguration.java:108)
at
org.apache.accumulo.core.file.DispatchingFileFactory.openReader(FileOperations.java:72)
at
org.apache.accumulo.server.tabletserver.Compactor.call(Compactor.java:150)
at
org.apache.accumulo.server.tabletserver.MinorCompactor.call(MinorCompactor.java:80)
at
org.apache.accumulo.server.tabletserver.Tablet.minorCompact(Tablet.java:2162)
at
org.apache.accumulo.server.tabletserver.Tablet.access$4400(Tablet.java:145)
at
org.apache.accumulo.server.tabletserver.Tablet$MinorCompactionTask.run(Tablet.java:2241)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,198 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/users/redact
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.security.ZKAuthenticator.authenticate(ZKAuthenticator.java:97)
at
org.apache.accumulo.server.security.ZKAuthenticator.authenticateUser(ZKAuthenticator.java:184)
at
org.apache.accumulo.server.security.Auditor.authenticateUser(Auditor.java:69)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startUpdate(TabletServer.java:1366)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at com.sun.proxy.$Proxy2.startUpdate(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startUpdate.process(TabletClientService.java:2294)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2053)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,198 [zookeeper.ZooLock] WARN : lost connection to zookeeper
2014-04-09 21:48:51,201 [zookeeper.DistributedWorkQueue] INFO : Got unexpected
zookeeper event: None for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/bulk_failed_copyq
2014-04-09 21:48:51,845 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/config/table.bloom.enabled
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.conf.ZooConfiguration.get(ZooConfiguration.java:109)
at
org.apache.accumulo.server.conf.ZooConfiguration._get(ZooConfiguration.java:76)
at
org.apache.accumulo.server.conf.ZooConfiguration.get(ZooConfiguration.java:103)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:114)
at
org.apache.accumulo.core.conf.AccumuloConfiguration.getBoolean(AccumuloConfiguration.java:108)
at
org.apache.accumulo.core.file.DispatchingFileFactory.openReader(FileOperations.java:72)
at
org.apache.accumulo.server.tabletserver.Compactor.call(Compactor.java:150)
at
org.apache.accumulo.server.tabletserver.MinorCompactor.call(MinorCompactor.java:80)
at
org.apache.accumulo.server.tabletserver.Tablet.minorCompact(Tablet.java:2162)
at
org.apache.accumulo.server.tabletserver.Tablet.access$4400(Tablet.java:145)
at
org.apache.accumulo.server.tabletserver.Tablet$MinorCompactionTask.run(Tablet.java:2241)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,845 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/users/redact
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.security.ZKAuthenticator.authenticate(ZKAuthenticator.java:97)
at
org.apache.accumulo.server.security.ZKAuthenticator.authenticateUser(ZKAuthenticator.java:184)
at
org.apache.accumulo.server.security.Auditor.authenticateUser(Auditor.java:69)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startUpdate(TabletServer.java:1366)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at com.sun.proxy.$Proxy2.startUpdate(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startUpdate.process(TabletClientService.java:2294)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2053)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,845 [zookeeper.ZooCache] WARN : Zookeeper error, will retry
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/tables/6/conf/table.bloom.enabled
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.accumulo.core.zookeeper.ZooCache$2.run(ZooCache.java:208)
at org.apache.accumulo.core.zookeeper.ZooCache.retry(ZooCache.java:130)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:233)
at org.apache.accumulo.core.zookeeper.ZooCache.get(ZooCache.java:188)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:121)
at
org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:109)
at
org.apache.accumulo.core.conf.AccumuloConfiguration.getBoolean(AccumuloConfiguration.java:108)
at
org.apache.accumulo.core.file.DispatchingFileFactory.openReader(FileOperations.java:72)
at
org.apache.accumulo.server.tabletserver.Compactor.call(Compactor.java:150)
at
org.apache.accumulo.server.tabletserver.MinorCompactor.call(MinorCompactor.java:80)
at
org.apache.accumulo.server.tabletserver.Tablet.minorCompact(Tablet.java:2162)
at
org.apache.accumulo.server.tabletserver.Tablet.access$4400(Tablet.java:145)
at
org.apache.accumulo.server.tabletserver.Tablet$MinorCompactionTask.run(Tablet.java:2241)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-09 21:48:51,866 [tabletserver.TabletServer] FATAL: Lost ability to
monitor tablet server lock, exiting.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/accumulo/ffabfdfb-15a7-41ec-9215-e686f90c4a36/tservers/10.128.199.83:9997/zlock-0000000000
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.accumulo.core.zookeeper.ZooReader.getStatus(ZooReader.java:55)
at
org.apache.accumulo.server.zookeeper.ZooLock$2.process(ZooLock.java:236)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
-bash-4.1$
