Do you have long running gc? I've seen this cause zk connection loss. On Aug 20, 2015 2:30 AM, "swapnil joshi" <[email protected]> wrote:
> Thanks!!! For giving me response. I had change configuration in storm.yml > file. But still I got following error > > *15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation > retry gave up* > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) > at > com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) > 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation > retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) > at > com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) > 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST > 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST > 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation > retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) > at > com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) > 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST > > > On Wed, Aug 19, 2015 at 8:16 PM, satyavrat <[email protected]> wrote: > >> Hi, >> Try to change worker timeout abd zookeeper timeout. >> Currently i am using below.. >> >> Configuration is : >> storm.zookeeper.session.timeout: 250000 >> storm.zookeeper.connection.timeout: 250000 >> nimbus.task.timeout.secs: 300 >> nimbus.monitor.freq.secs: 10 >> storm.local.dir: "/mnt/storm" >> supervisor.worker.timeout.secs: 250 >> worker.heartbeat.frequency.secs: 5 >> supervisor.monitor.frequency.secs: 10 >> supervisor.heartbeat.frequency.secs: 5 >> >> - satya >> On Aug 19, 2015 8:12 PM, "swapnil joshi" <[email protected]> >> wrote: >> >>> Hi Friends, >>> >>> In my topology one bolt required time to prepare. Because, It's >>> doing lots of data loading operation before preparing this bolt storm gives >>> me following error. >>> >>> *server.ZooKeeperServer: Expiring session 0x14f4657e8dd001b, timeout of >>> 20000ms exceeded* >>> >>> What is problem? >>> How will I increase zookeeper Timeout? >>> >>> >>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>> 0x14f4657e8dd001b, timeout of 20000ms exceeded >>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>> 0x14f4657e8dd001a, timeout of 20000ms exceeded >>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>> 0x14f4657e8dd001e, timeout of 20000ms exceeded >>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>> 0x14f4657e8dd0019, timeout of 20000ms exceeded >>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>> termination for sessionid: 0x14f4657e8dd001b >>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>> 0x14f4657e8dd001d, timeout of 20000ms exceeded >>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>> 0x14f4657e8dd001c, timeout of 20000ms exceeded >>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>> termination for sessionid: 0x14f4657e8dd001a >>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>> termination for sessionid: 0x14f4657e8dd001e >>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>> termination for sessionid: 0x14f4657e8dd0019 >>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>> termination for sessionid: 0x14f4657e8dd001d >>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>> termination for sessionid: 0x14f4657e8dd001c >>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>> Background operation retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>> Background operation retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>> ConnectionStateListeners registered. >>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>> ConnectionStateListeners registered. >>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>> Background operation retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>> ConnectionStateListeners registered. >>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>> ConnectionStateListeners registered. >>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>> Background operation retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>> ConnectionStateListeners registered. >>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>> Background operation retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>> ConnectionStateListeners registered. >>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>> Background operation retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/19 20:05:06 INFO storm.util: Halting process: ("Unrecoverable >>> Zookeeper error") >>> >>> -- >>> Regards, >>> Swapnil K. Joshi >>> >> > > > -- > Regards, > Swapnil K. Joshi >
