What’s your OS?
On 3/27/14, 9:42 PM, "李家宏" <[email protected]> wrote:
>Andy,
>
>Sorry for the late reply. Here is the stack trace:
>
>2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport
>plugin:backtype.storm.messaging.netty.Context
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of
>server mk-worker
> org.jboss.netty.channel.ChannelException: Failed to create a selector.
> at
>org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
>ctNioSelector.java:337)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelecto
>r.java:95)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.ja
>va:51)
>~[netty-3.6.3.Final.jar:na]
> at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
>l.java:45)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
>l.java:28)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(Abstrac
>tNioWorkerPool.java:99)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioW
>orkerPool.java:69)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
>tSocketChannelFactory.java:152)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
>tSocketChannelFactory.java:134)
> ~[netty-3.6.3.Final.jar:na]
> at backtype.storm.messaging.netty.Client.(Client.java:54)
>~[storm-netty-0.9.0.1.jar:na]
> at backtype.storm.messaging.netty.Context.connect(Context.java:36)
>~[storm-netty-0.9.0.1.jar:na]
> at
>backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834_
>_5838$fn__5839.invoke(worker.clj:250)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
> at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na]
> at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na]
> at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na]
> at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na]
> at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
> at
>backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(work
>er.clj:244)
>~[storm-core-0.9.0.1.jar:na]
> at
>backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(w
>orker.clj:357)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
> at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
> at
>backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj:
>329)
>[storm-core-0.9.0.1.jar:na]
> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
> at backtype.storm.daemon.worker$_main.invoke(worker.clj:439)
>[storm-core-0.9.0.1.jar:na]
> at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
> at backtype.storm.daemon.worker.main(Unknown Source)
>[storm-core-0.9.0.1.jar:na]
>
> * Caused by: java.io.IOException: Too many open files*
>
> at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38]
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
>~[na:1.6.0_38]
> at
>sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:1
>8)
>~[na:1.6.0_38]
> at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
>ctNioSelector.java:335)
> ~[netty-3.6.3.Final.jar:na]
> ... 32 common frames omitted
> 2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on
>initialization")
>--------------------------------------------------------------------------
>------------------------------------------
>
>Regards
>
>
>2014-03-12 3:25 GMT+08:00 Andrew Feng <[email protected]>:
>
>> Gavin:
>>
>> In our production, we have seen situation that Netty implementation uses
>> too many threads.
>> It uses a dedicated thread for each remote bolt worker. We have a fix to
>> avoid that problem.
>>
>>
>> Our problem could be different, since we did not face
>> too-many-file-descriptor issue.
>> Can you post the stack trace?
>>
>> Andy
>>
>> On 3/5/14 1:26 AM, "李家宏" <[email protected]> wrote:
>>
>> >Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
>> >message transport layer.
>> >I guess netty must be the problem.
>> >
>> >Regards
>> >
>> >
>> >2014-03-05 10:56 GMT+08:00 李家宏 <[email protected]>:
>> >
>> >> Hi, all
>> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as
>> >>transport
>> >> layer, this error occurs :
>> >> Netty client failed to create a selector due to* too many open files
>> >> exception*, the worker continuously halting with initialization
>>error.
>> >>
>> >> I checked the ulimit -n(> 130000) which is much bigger than currently
>> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at
>>most.
>> >>
>> >> By the way,this topology works fine with storm cluster of 0.8.0.
>> >>
>> >> What's the problem?
>> >>
>> >> Thanks
>> >>
>> >> --
>> >>
>> >> ======================================================
>> >>
>> >> Gvain
>> >>
>> >> Email: [email protected]
>> >>
>> >
>> >
>> >
>> >--
>> >
>> >======================================================
>> >
>> >Gvain
>> >
>> >Email: [email protected]
>>
>>
>
>
>--
>
>======================================================
>
>Gvain
>
>Email: [email protected]
