In this case you probably need more hardware... either more RAM or another machine. Really if you have any sort of somewhat serious deadlines you don't want to spend > 20 s garbage collecting, and you don't want to be depending on swap for half of your application's memory.
On Fri, Aug 21, 2015 at 7:36 AM, swapnil joshi <[email protected]> wrote: > Yes, You are right. This is GC problem. Now to avoid this issue. I have > increased my JVM Heap size. Now the new problem rises. After some time bolt > start prepare method again and again. > > Currently, My System have 16GB RAM and SWAP area is Approx. 28GB. And As > per basic calculation, I need 26GB memory. I have run this topology in > local cluster mode. > > > On Fri, Aug 21, 2015 at 12:18 AM, Nathan Leung <[email protected]> wrote: > >> Prepare shouldn't cause a timeout. When I said gc I meant something like >> stop the world garbage collection. >> >> On Thu, Aug 20, 2015 at 6:52 AM, swapnil joshi < >> [email protected]> wrote: >> >>> Hi Nathan, >>> Yes, One of my bolt takes lot of time to prepare because it contains >>> some data preparation Operation. So what are solutions for such problem. >>> >>> On Thu, Aug 20, 2015 at 4:19 PM, Nathan Leung <[email protected]> wrote: >>> >>>> Do you have long running gc? I've seen this cause zk connection loss. >>>> On Aug 20, 2015 2:30 AM, "swapnil joshi" <[email protected]> >>>> wrote: >>>> >>>>> Thanks!!! For giving me response. I had change configuration in >>>>> storm.yml file. But still I got following error >>>>> >>>>> *15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up* >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >>>>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background >>>>> operation retry gave up >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> KeeperErrorCode = ConnectionLoss >>>>> at >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>> at >>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>> at >>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>> at >>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >>>>> >>>>> >>>>> On Wed, Aug 19, 2015 at 8:16 PM, satyavrat <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> Try to change worker timeout abd zookeeper timeout. >>>>>> Currently i am using below.. >>>>>> >>>>>> Configuration is : >>>>>> storm.zookeeper.session.timeout: 250000 >>>>>> storm.zookeeper.connection.timeout: 250000 >>>>>> nimbus.task.timeout.secs: 300 >>>>>> nimbus.monitor.freq.secs: 10 >>>>>> storm.local.dir: "/mnt/storm" >>>>>> supervisor.worker.timeout.secs: 250 >>>>>> worker.heartbeat.frequency.secs: 5 >>>>>> supervisor.monitor.frequency.secs: 10 >>>>>> supervisor.heartbeat.frequency.secs: 5 >>>>>> >>>>>> - satya >>>>>> On Aug 19, 2015 8:12 PM, "swapnil joshi" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Friends, >>>>>>> >>>>>>> In my topology one bolt required time to prepare. Because, It's >>>>>>> doing lots of data loading operation before preparing this bolt storm >>>>>>> gives >>>>>>> me following error. >>>>>>> >>>>>>> *server.ZooKeeperServer: Expiring session 0x14f4657e8dd001b, timeout >>>>>>> of 20000ms exceeded* >>>>>>> >>>>>>> What is problem? >>>>>>> How will I increase zookeeper Timeout? >>>>>>> >>>>>>> >>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>> 0x14f4657e8dd001b, timeout of 20000ms exceeded >>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>> 0x14f4657e8dd001a, timeout of 20000ms exceeded >>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>> 0x14f4657e8dd001e, timeout of 20000ms exceeded >>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>> 0x14f4657e8dd0019, timeout of 20000ms exceeded >>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>> session termination for sessionid: 0x14f4657e8dd001b >>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>> 0x14f4657e8dd001d, timeout of 20000ms exceeded >>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>> 0x14f4657e8dd001c, timeout of 20000ms exceeded >>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>> session termination for sessionid: 0x14f4657e8dd001a >>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>> session termination for sessionid: 0x14f4657e8dd001e >>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>> session termination for sessionid: 0x14f4657e8dd0019 >>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>> session termination for sessionid: 0x14f4657e8dd001d >>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>> session termination for sessionid: 0x14f4657e8dd001c >>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>> operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>> operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>> LOST >>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>> operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>> operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>> LOST >>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>> operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>> operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>> error Background operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>> error Background operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>> LOST >>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>> ConnectionStateListeners registered. >>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>> LOST >>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>> ConnectionStateListeners registered. >>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>> error Background operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>> LOST >>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>> ConnectionStateListeners registered. >>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>> ConnectionStateListeners registered. >>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>> error Background operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>> LOST >>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>> ConnectionStateListeners registered. >>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>> error Background operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>> ConnectionStateListeners registered. >>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>> error Background operation retry gave up >>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>> at >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>> at >>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>> at >>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>> 15/08/19 20:05:06 INFO storm.util: Halting process: ("Unrecoverable >>>>>>> Zookeeper error") >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Swapnil K. Joshi >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Swapnil K. Joshi >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Swapnil K. Joshi >>> >> >> > > > -- > Regards, > Swapnil K. Joshi >
