I t seems you created too many threads, that cause no available thread resource.
Sent from my iPhone > On Jun 12, 2016, at 10:18, "[email protected]" <[email protected]> > wrote: > > Hi user, > > I have a topology to write into HBase. Every time I submitted the topology, > it runned well. But after a well, for example, one or two days, the topology > always reports an execption like below: > > java.lang.OutOfMemoryError: unable to create new native thread at > java.lang.Thread.start0(Native Method) at > java.lang.Thread.start(Thread.java:714) at > org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134) > at > org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179) > at > org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153) > at > org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135) > at > org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234) > at > org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306) at > com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22) > at com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279) > at > backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) > at > backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) > at > backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > at > backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) > at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at > clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745) > then the topology runs very slow. > I took a look at the log, it is full of the information like below: > 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process > identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper > ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection, > connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, > quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to > server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using > SASL (unknown error) > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established to > r720m8-hdp/10.6.116.3:2181, initiating session > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete on > server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f, negotiated > timeout = 120000 > 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f closed > 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down > 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process > identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper > ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection, > connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 > sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, > quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to > server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using > SASL (unknown error) > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established to > r720m8-hdp/10.6.116.3:2181, initiating session > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete on > server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f, negotiated > timeout = 120000 > 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f closed > 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down > > Anyone comes across this problem? Thanks for your hints. > > Joshua > 2016-06-12 10:12:33 > >
