Prepare shouldn't cause a timeout. When I said gc I meant something like stop the world garbage collection.
On Thu, Aug 20, 2015 at 6:52 AM, swapnil joshi <[email protected]> wrote: > Hi Nathan, > Yes, One of my bolt takes lot of time to prepare because it contains some > data preparation Operation. So what are solutions for such problem. > > On Thu, Aug 20, 2015 at 4:19 PM, Nathan Leung <[email protected]> wrote: > >> Do you have long running gc? I've seen this cause zk connection loss. >> On Aug 20, 2015 2:30 AM, "swapnil joshi" <[email protected]> >> wrote: >> >>> Thanks!!! For giving me response. I had change configuration in >>> storm.yml file. But still I got following error >>> >>> *15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up* >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation >>> retry gave up >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> at >>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>> at >>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>> at >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >>> >>> >>> On Wed, Aug 19, 2015 at 8:16 PM, satyavrat <[email protected]> wrote: >>> >>>> Hi, >>>> Try to change worker timeout abd zookeeper timeout. >>>> Currently i am using below.. >>>> >>>> Configuration is : >>>> storm.zookeeper.session.timeout: 250000 >>>> storm.zookeeper.connection.timeout: 250000 >>>> nimbus.task.timeout.secs: 300 >>>> nimbus.monitor.freq.secs: 10 >>>> storm.local.dir: "/mnt/storm" >>>> supervisor.worker.timeout.secs: 250 >>>> worker.heartbeat.frequency.secs: 5 >>>> supervisor.monitor.frequency.secs: 10 >>>> supervisor.heartbeat.frequency.secs: 5 >>>> >>>> - satya >>>> On Aug 19, 2015 8:12 PM, "swapnil joshi" <[email protected]> >>>> wrote: >>>> >>>>> Hi Friends, >>>>> >>>>> In my topology one bolt required time to prepare. Because, It's >>>>> doing lots of data loading operation before preparing this bolt storm >>>>> gives >>>>> me following error. >>>>> >>>>> *server.ZooKeeperServer: Expiring session 0x14f4657e8dd001b, timeout >>>>> of 20000ms exceeded* >>>>> >>>>> What is problem? >>>>> How will I increase zookeeper Timeout? >>>>> >>>>> >>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>> 0x14f4657e8dd001b, timeout of 20000ms exceeded >>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>> 0x14f4657e8dd001a, timeout of 20000ms exceeded >>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>> 0x14f4657e8dd001e, timeout of 20000ms exceeded >>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>> 0x14f4657e8dd0019, timeout of 20000ms exceeded >>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>>> termination for sessionid: 0x14f4657e8dd001b >>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>> 0x14f4657e8dd001d, timeout of 20000ms exceeded >>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>> 0x14f4657e8dd001c, timeout of 20000ms exceeded >>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>>> termination for sessionid: 0x14f4657e8dd001a >>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>>> termination for sessionid: 0x14f4657e8dd001e >>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>>> termination for sessionid: 0x14f4657e8dd0019 >>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>>> termination for sessionid: 0x14f4657e8dd001d >>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>>> termination for sessionid: 0x14f4657e8dd001c >>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>>> Background operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>>> Background operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>> ConnectionStateListeners registered. >>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>> ConnectionStateListeners registered. >>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>>> Background operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>> ConnectionStateListeners registered. >>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>> ConnectionStateListeners registered. >>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>>> Background operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>> ConnectionStateListeners registered. >>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>>> Background operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>> ConnectionStateListeners registered. >>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>>> Background operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/19 20:05:06 INFO storm.util: Halting process: ("Unrecoverable >>>>> Zookeeper error") >>>>> >>>>> -- >>>>> Regards, >>>>> Swapnil K. Joshi >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Swapnil K. Joshi >>> >> > > > -- > Regards, > Swapnil K. Joshi >
