Simply reinstalled Ubuntu and now it works fine. :D > Did you ever find the cause?
As I attached the link, it look like a issue on Netty side. BTW, I feel like I need to explain my experiment here. Few weeks ago, I committed HAMA-954 so that you can turn on/off the message bundle behavior. If you set hama.bsp.messenger.bundle to true, the messenger system package the messages into single BSPMessageBundle object and send it at once. Meanwhile, if you set hama.bsp.messenger.bundle to false, the system send a message one by one asynchronously. I think the advantage of netty's non-blocking RPC can be maximized in this case (I couldn't test it yet). -- Best Regards, Edward J. Yoon -----Original Message----- From: ByungSeok Min [mailto:[email protected]] Sent: Friday, May 08, 2015 5:15 PM To: [email protected] Subject: Re: Bug in Netty-based RPC Hi~ Mr Yoon. Is your machine ok? Did you ever find the cause? * My Work in progress is going to be finished well. * I'll be back soon. Best Regards! 2015-04-28 12:58 GMT+09:00 Edward J. Yoon <[email protected]>: > I just found this > > https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324 > > On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin <[email protected]> > wrote: > > Have you checked limit.conf? > > > > From the message it looks like the files opened at underlying system > > exceed its default limit. > > > > On 28 April 2015 at 08:08, Edward J. Yoon <[email protected]> wrote: > >> I tried to run BSP job using netty-based RPC instead of message > >> bundle, but I received "too many open files". > >> > >> -- > >> > >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO > >> ipc.AsyncClient: AsyncClient startup > >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR > >> bsp.BSPTask: Error running bsp setup and bsp function. > >> attempt_201504280858_0001_000017_0: java.lang.IllegalStateException: > >> failed to create a child event loop > >> attempt_201504280858_0001_000017_0: at > >> > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68) > >> attempt_201504280858_0001_000017_0: at > >> > io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49) > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61) > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52) > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44) > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) > >> attempt_201504280858_0001_000017_0: at > >> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) > >> attempt_201504280858_0001_000017_0: at > >> > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) > >> attempt_201504280858_0001_000017_0: at > >> > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) > >> attempt_201504280858_0001_000017_0: at > >> > org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > >> attempt_201504280858_0001_000017_0: at > >> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > >> attempt_201504280858_0001_000017_0: Caused by: > >> io.netty.channel.ChannelException: failed to open a new selector > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128) > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120) > >> attempt_201504280858_0001_000017_0: at > >> > io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87) > >> attempt_201504280858_0001_000017_0: at > >> > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64) > >> attempt_201504280858_0001_000017_0: ... 24 more > >> attempt_201504280858_0001_000017_0: Caused by: java.io.IOException: > >> Too many open files > >> attempt_201504280858_0001_000017_0: at > sun.nio.ch.IOUtil.makePipe(Native Method) > >> attempt_201504280858_0001_000017_0: at > >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) > >> attempt_201504280858_0001_000017_0: at > >> > sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) > >> attempt_201504280858_0001_000017_0: at > >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126) > >> attempt_201504280858_0001_000017_0: ... 27 more > >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO > >> ipc.AsyncServer: AsyncServer gracefully shutdown > >> > >> > >> -- > >> Best Regards, Edward J. Yoon > > > > -- > Best Regards, Edward J. Yoon >
