Hi, Your message says it is throwing OutOfMemory error. So, you should look into what is causing that. It may not be really because of storm but it may be because of application code also. You may want to use ` -XX:+HeapDumpOnOutOfMemoryError` and `-XX:HeapDumpPath=/worker/dumps` to dump heap on outofmemory and analyze what is causing OutOfMemory.
Thanks, Satish. On Sun, Jun 12, 2016 at 11:34 AM, [email protected] < [email protected]> wrote: > Hi Wenwei > > Actually, I did not new any thread in my bolt. The error reported just > comes from storm core itself. > > ------------------------------ > > I t seems you created too many threads, that > cause no available thread resource. > > Sent from my iPhone > > On Jun 12, 2016, at 10:18, "[email protected]" < > [email protected]> wrote: > > *Hi user,* > > *I have a topology to write into HBase. Every time I submitted the > topology, it runned well. But after a well, for example, one or two days, > the topology always reports an execption like below:* > > java.lang.OutOfMemoryError: unable to create new native thread at > java.lang.Thread.start0(Native Method) at > java.lang.Thread.start(Thread.java:714) at > org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134) > at > org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179) > at > org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153) > at > org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135) > at > org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234) > at > org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306) > at > com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22) > at > com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279) > at > backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) > at > backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) > at > backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > at > backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) > at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at > clojure.lang.AFn.run(AFn.java:24) at > java.lang.Thread.run(Thread.java:745)*then > the topology runs very slow.* > *I took a look at the log, it is full of the information like below:* > 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process > identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper > ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection, > connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, > quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to > server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using > SASL (unknown error) > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established > to r720m8-hdp/10.6.116.3:2181, initiating session > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete > on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f, > negotiated timeout = 120000 > 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f > closed > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down > 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process > identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper > ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection, > connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, > quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to > server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using > SASL (unknown error) > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established > to r720m8-hdp/10.6.116.3:2181, initiating session > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete > on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f, > negotiated timeout = 120000 > 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f > closed > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down > > *Anyone comes across this problem? Thanks for your hints.* > > ------------------------------ > Joshua > 2016-06-12 10:12:33 > > >
