Red Hat Enterprise Linux Server release 6.2 (Santiago)
2014-03-28 12:55 GMT+08:00 Andy Feng <[email protected]>: > What’s your OS? > > On 3/27/14, 9:42 PM, "李家宏" <[email protected]> wrote: > > >Andy, > > > >Sorry for the late reply. Here is the stack trace: > > > >2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport > >plugin:backtype.storm.messaging.netty.Context > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] > > 2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of > >server mk-worker > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at > >org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra > >ctNioSelector.java:337) > > ~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelecto > >r.java:95) > >~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.ja > >va:51) > >~[netty-3.6.3.Final.jar:na] > > at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45) > >~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo > >l.java:45) > >~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo > >l.java:28) > >~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(Abstrac > >tNioWorkerPool.java:99) > > ~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioW > >orkerPool.java:69) > > ~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39) > >~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33) > >~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien > >tSocketChannelFactory.java:152) > > ~[netty-3.6.3.Final.jar:na] > > at > >org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien > >tSocketChannelFactory.java:134) > > ~[netty-3.6.3.Final.jar:na] > > at backtype.storm.messaging.netty.Client.(Client.java:54) > >~[storm-netty-0.9.0.1.jar:na] > > at backtype.storm.messaging.netty.Context.connect(Context.java:36) > >~[storm-netty-0.9.0.1.jar:na] > > at > >backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834_ > >_5838$fn__5839.invoke(worker.clj:250) > > ~[storm-core-0.9.0.1.jar:na] > > at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na] > > at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na] > > at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na] > > at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na] > > at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na] > > at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na] > > at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na] > > at > >backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(work > >er.clj:244) > >~[storm-core-0.9.0.1.jar:na] > > at > >backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(w > >orker.clj:357) > > ~[storm-core-0.9.0.1.jar:na] > > at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] > > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] > > at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] > > at > >backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj: > >329) > >[storm-core-0.9.0.1.jar:na] > > at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] > > at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) > >[storm-core-0.9.0.1.jar:na] > > at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] > > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] > > at backtype.storm.daemon.worker.main(Unknown Source) > >[storm-core-0.9.0.1.jar:na] > > > > * Caused by: java.io.IOException: Too many open files* > > > > at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38] > > at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49) > >~[na:1.6.0_38] > > at > >sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:1 > >8) > >~[na:1.6.0_38] > > at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38] > > at > >org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra > >ctNioSelector.java:335) > > ~[netty-3.6.3.Final.jar:na] > > ... 32 common frames omitted > > 2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on > >initialization") > >-------------------------------------------------------------------------- > >------------------------------------------ > > > >Regards > > > > > >2014-03-12 3:25 GMT+08:00 Andrew Feng <[email protected]>: > > > >> Gavin: > >> > >> In our production, we have seen situation that Netty implementation uses > >> too many threads. > >> It uses a dedicated thread for each remote bolt worker. We have a fix to > >> avoid that problem. > >> > >> > >> Our problem could be different, since we did not face > >> too-many-file-descriptor issue. > >> Can you post the stack trace? > >> > >> Andy > >> > >> On 3/5/14 1:26 AM, "李家宏" <[email protected]> wrote: > >> > >> >Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as > >> >message transport layer. > >> >I guess netty must be the problem. > >> > > >> >Regards > >> > > >> > > >> >2014-03-05 10:56 GMT+08:00 李家宏 <[email protected]>: > >> > > >> >> Hi, all > >> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as > >> >>transport > >> >> layer, this error occurs : > >> >> Netty client failed to create a selector due to* too many open files > >> >> exception*, the worker continuously halting with initialization > >>error. > >> >> > >> >> I checked the ulimit -n(> 130000) which is much bigger than currently > >> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at > >>most. > >> >> > >> >> By the way,this topology works fine with storm cluster of 0.8.0. > >> >> > >> >> What's the problem? > >> >> > >> >> Thanks > >> >> > >> >> -- > >> >> > >> >> ====================================================== > >> >> > >> >> Gvain > >> >> > >> >> Email: [email protected] > >> >> > >> > > >> > > >> > > >> >-- > >> > > >> >====================================================== > >> > > >> >Gvain > >> > > >> >Email: [email protected] > >> > >> > > > > > >-- > > > >====================================================== > > > >Gvain > > > >Email: [email protected] > > > -- ====================================================== Gvain Email: [email protected]
