Thank You Nathan!!! On Fri, Aug 21, 2015 at 5:51 PM, Nathan Leung <[email protected]> wrote:
> In this case you probably need more hardware... either more RAM or another > machine. Really if you have any sort of somewhat serious deadlines you > don't want to spend > 20 s garbage collecting, and you don't want to be > depending on swap for half of your application's memory. > > On Fri, Aug 21, 2015 at 7:36 AM, swapnil joshi < > [email protected]> wrote: > >> Yes, You are right. This is GC problem. Now to avoid this issue. I have >> increased my JVM Heap size. Now the new problem rises. After some time bolt >> start prepare method again and again. >> >> Currently, My System have 16GB RAM and SWAP area is Approx. 28GB. And As >> per basic calculation, I need 26GB memory. I have run this topology in >> local cluster mode. >> >> >> On Fri, Aug 21, 2015 at 12:18 AM, Nathan Leung <[email protected]> wrote: >> >>> Prepare shouldn't cause a timeout. When I said gc I meant something >>> like stop the world garbage collection. >>> >>> On Thu, Aug 20, 2015 at 6:52 AM, swapnil joshi < >>> [email protected]> wrote: >>> >>>> Hi Nathan, >>>> Yes, One of my bolt takes lot of time to prepare because it contains >>>> some data preparation Operation. So what are solutions for such problem. >>>> >>>> On Thu, Aug 20, 2015 at 4:19 PM, Nathan Leung <[email protected]> >>>> wrote: >>>> >>>>> Do you have long running gc? I've seen this cause zk connection loss. >>>>> On Aug 20, 2015 2:30 AM, "swapnil joshi" <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks!!! For giving me response. I had change configuration in >>>>>> storm.yml file. But still I got following error >>>>>> >>>>>> *15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background >>>>>> operation retry gave up* >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>> KeeperErrorCode = ConnectionLoss >>>>>> at >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>> at >>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>> at >>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>> at >>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>> at >>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background >>>>>> operation retry gave up >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>> KeeperErrorCode = ConnectionLoss >>>>>> at >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>> at >>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>> at >>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>> at >>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>> at >>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: >>>>>> LOST >>>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: >>>>>> LOST >>>>>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background >>>>>> operation retry gave up >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>> KeeperErrorCode = ConnectionLoss >>>>>> at >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>> at >>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>> at >>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>> at >>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>> at >>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: >>>>>> LOST >>>>>> >>>>>> >>>>>> On Wed, Aug 19, 2015 at 8:16 PM, satyavrat <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> Try to change worker timeout abd zookeeper timeout. >>>>>>> Currently i am using below.. >>>>>>> >>>>>>> Configuration is : >>>>>>> storm.zookeeper.session.timeout: 250000 >>>>>>> storm.zookeeper.connection.timeout: 250000 >>>>>>> nimbus.task.timeout.secs: 300 >>>>>>> nimbus.monitor.freq.secs: 10 >>>>>>> storm.local.dir: "/mnt/storm" >>>>>>> supervisor.worker.timeout.secs: 250 >>>>>>> worker.heartbeat.frequency.secs: 5 >>>>>>> supervisor.monitor.frequency.secs: 10 >>>>>>> supervisor.heartbeat.frequency.secs: 5 >>>>>>> >>>>>>> - satya >>>>>>> On Aug 19, 2015 8:12 PM, "swapnil joshi" < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Friends, >>>>>>>> >>>>>>>> In my topology one bolt required time to prepare. Because, It's >>>>>>>> doing lots of data loading operation before preparing this bolt storm >>>>>>>> gives >>>>>>>> me following error. >>>>>>>> >>>>>>>> *server.ZooKeeperServer: Expiring session 0x14f4657e8dd001b, >>>>>>>> timeout of 20000ms exceeded* >>>>>>>> >>>>>>>> What is problem? >>>>>>>> How will I increase zookeeper Timeout? >>>>>>>> >>>>>>>> >>>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>>> 0x14f4657e8dd001b, timeout of 20000ms exceeded >>>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>>> 0x14f4657e8dd001a, timeout of 20000ms exceeded >>>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>>> 0x14f4657e8dd001e, timeout of 20000ms exceeded >>>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>>> 0x14f4657e8dd0019, timeout of 20000ms exceeded >>>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>>> session termination for sessionid: 0x14f4657e8dd001b >>>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>>> 0x14f4657e8dd001d, timeout of 20000ms exceeded >>>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>>>>>> 0x14f4657e8dd001c, timeout of 20000ms exceeded >>>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>>> session termination for sessionid: 0x14f4657e8dd001a >>>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>>> session termination for sessionid: 0x14f4657e8dd001e >>>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>>> session termination for sessionid: 0x14f4657e8dd0019 >>>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>>> session termination for sessionid: 0x14f4657e8dd001d >>>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed >>>>>>>> session termination for sessionid: 0x14f4657e8dd001c >>>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>>> operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>>> operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>>> LOST >>>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>>> operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>>> operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>>> LOST >>>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>>> operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background >>>>>>>> operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>>> error Background operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>>> error Background operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>>> LOST >>>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>>> ConnectionStateListeners registered. >>>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>>> LOST >>>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>>> ConnectionStateListeners registered. >>>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>>> error Background operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>>> LOST >>>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>>> ConnectionStateListeners registered. >>>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>>> ConnectionStateListeners registered. >>>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>>> error Background operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: >>>>>>>> LOST >>>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>>> ConnectionStateListeners registered. >>>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>>> error Background operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>>>>>> ConnectionStateListeners registered. >>>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper >>>>>>>> error Background operation retry gave up >>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>>>>> KeeperErrorCode = ConnectionLoss >>>>>>>> at >>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>>>>>> at >>>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>>>>>> at >>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>>>>>> 15/08/19 20:05:06 INFO storm.util: Halting process: ("Unrecoverable >>>>>>>> Zookeeper error") >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Swapnil K. Joshi >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Swapnil K. Joshi >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Swapnil K. Joshi >>>> >>> >>> >> >> >> -- >> Regards, >> Swapnil K. Joshi >> > > -- Regards, Swapnil K. Joshi
