RE: Bug in Netty-based RPC
Simply reinstalled Ubuntu and now it works fine. :D > Did you ever find the cause? As I attached the link, it look like a issue on Netty side. BTW, I feel like I need to explain my experiment here. Few weeks ago, I committed HAMA-954 so that you can turn on/off the message bundle behavior. If you set hama.bsp.messenger.bundle to true, the messenger system package the messages into single BSPMessageBundle object and send it at once. Meanwhile, if you set hama.bsp.messenger.bundle to false, the system send a message one by one asynchronously. I think the advantage of netty's non-blocking RPC can be maximized in this case (I couldn't test it yet). -- Best Regards, Edward J. Yoon -Original Message- From: ByungSeok Min [mailto:byeongseok@gmail.com] Sent: Friday, May 08, 2015 5:15 PM To: dev@hama.apache.org Subject: Re: Bug in Netty-based RPC Hi~ Mr Yoon. Is your machine ok? Did you ever find the cause? * My Work in progress is going to be finished well. * I'll be back soon. Best Regards! 2015-04-28 12:58 GMT+09:00 Edward J. Yoon : > I just found this > > https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324 > > On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin > wrote: > > Have you checked limit.conf? > > > > From the message it looks like the files opened at underlying system > > exceed its default limit. > > > > On 28 April 2015 at 08:08, Edward J. Yoon wrote: > >> I tried to run BSP job using netty-based RPC instead of message > >> bundle, but I received "too many open files". > >> > >> -- > >> > >> attempt_201504280858_0001_17_0: 15/04/28 08:28:17 INFO > >> ipc.AsyncClient: AsyncClient startup > >> attempt_201504280858_0001_17_0: 15/04/28 08:28:21 ERROR > >> bsp.BSPTask: Error running bsp setup and bsp function. > >> attempt_201504280858_0001_17_0: java.lang.IllegalStateException: > >> failed to create a child event loop > >> attempt_201504280858_0001_17_0: at > >> > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68) > >> attempt_201504280858_0001_17_0: at > >> > io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncClient$Connection.(AsyncClient.java:189) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) > >> attempt_201504280858_0001_17_0: at > >> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) > >> attempt_201504280858_0001_17_0: at > >> > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) > >> attempt_201504280858_0001_17_0: at > >> > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) > >> attempt_201504280858_0001_17_0: at > >> > org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunn
Re: Bug in Netty-based RPC
Hi~ Mr Yoon. Is your machine ok? Did you ever find the cause? * My Work in progress is going to be finished well. * I'll be back soon. Best Regards! 2015-04-28 12:58 GMT+09:00 Edward J. Yoon : > I just found this > > https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324 > > On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin > wrote: > > Have you checked limit.conf? > > > > From the message it looks like the files opened at underlying system > > exceed its default limit. > > > > On 28 April 2015 at 08:08, Edward J. Yoon wrote: > >> I tried to run BSP job using netty-based RPC instead of message > >> bundle, but I received "too many open files". > >> > >> -- > >> > >> attempt_201504280858_0001_17_0: 15/04/28 08:28:17 INFO > >> ipc.AsyncClient: AsyncClient startup > >> attempt_201504280858_0001_17_0: 15/04/28 08:28:21 ERROR > >> bsp.BSPTask: Error running bsp setup and bsp function. > >> attempt_201504280858_0001_17_0: java.lang.IllegalStateException: > >> failed to create a child event loop > >> attempt_201504280858_0001_17_0: at > >> > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68) > >> attempt_201504280858_0001_17_0: at > >> > io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncClient$Connection.(AsyncClient.java:189) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) > >> attempt_201504280858_0001_17_0: at > >> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) > >> attempt_201504280858_0001_17_0: at > >> > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) > >> attempt_201504280858_0001_17_0: at > >> > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) > >> attempt_201504280858_0001_17_0: at > >> > org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > >> attempt_201504280858_0001_17_0: at > >> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > >> attempt_201504280858_0001_17_0: Caused by: > >> io.netty.channel.ChannelException: failed to open a new selector > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128) > >> attempt_201504280858_0001_17_0: at > >> io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120) > >> attempt_201504280858_0001_17_0: at > >> > io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87) > >> attempt_201504280858_0001_17_0: at > >> > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64) > >> attempt_201504280858_0001_17_0: ... 24 more > >> attempt_201504280858_0001_17_0: Caused by: java.io.IOException: > >> Too many open files > >> attempt_201504280858_0001_17_0: at > sun.nio.ch.IOUtil.makePipe(Native Method) > >> attempt_201504280858_0001_17_0: at > >> sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65) > >> attempt_201504280858_0001_17_0: at
Re: Bug in Netty-based RPC
wow, my machine has been crashed. On Tue, Apr 28, 2015 at 12:58 PM, Edward J. Yoon wrote: > I just found this > https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324 > > On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin wrote: >> Have you checked limit.conf? >> >> From the message it looks like the files opened at underlying system >> exceed its default limit. >> >> On 28 April 2015 at 08:08, Edward J. Yoon wrote: >>> I tried to run BSP job using netty-based RPC instead of message >>> bundle, but I received "too many open files". >>> >>> -- >>> >>> attempt_201504280858_0001_17_0: 15/04/28 08:28:17 INFO >>> ipc.AsyncClient: AsyncClient startup >>> attempt_201504280858_0001_17_0: 15/04/28 08:28:21 ERROR >>> bsp.BSPTask: Error running bsp setup and bsp function. >>> attempt_201504280858_0001_17_0: java.lang.IllegalStateException: >>> failed to create a child event loop >>> attempt_201504280858_0001_17_0: at >>> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncClient$Connection.(AsyncClient.java:189) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) >>> attempt_201504280858_0001_17_0: at >>> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) >>> attempt_201504280858_0001_17_0: at >>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) >>> attempt_201504280858_0001_17_0: Caused by: >>> io.netty.channel.ChannelException: failed to open a new selector >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87) >>> attempt_201504280858_0001_17_0: at >>> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64) >>> attempt_201504280858_0001_17_0: ... 24 more >>> attempt_201504280858_0001_17_0: Caused by: java.io.IOException: >>> Too many open files >>> attempt_201504280858_0001_17_0: at sun.nio.ch.IOUtil.makePipe(Native >>> Method) >>> attempt_201504280858_0001_17_0: at >>> sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65) >>> attempt_201504280858_0001_17_0: at >>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) >>> attempt_201504280858_0001_17_0: at >>> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126) >>> attem
Re: Bug in Netty-based RPC
I just found this https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324 On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin wrote: > Have you checked limit.conf? > > From the message it looks like the files opened at underlying system > exceed its default limit. > > On 28 April 2015 at 08:08, Edward J. Yoon wrote: >> I tried to run BSP job using netty-based RPC instead of message >> bundle, but I received "too many open files". >> >> -- >> >> attempt_201504280858_0001_17_0: 15/04/28 08:28:17 INFO >> ipc.AsyncClient: AsyncClient startup >> attempt_201504280858_0001_17_0: 15/04/28 08:28:21 ERROR >> bsp.BSPTask: Error running bsp setup and bsp function. >> attempt_201504280858_0001_17_0: java.lang.IllegalStateException: >> failed to create a child event loop >> attempt_201504280858_0001_17_0: at >> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncClient$Connection.(AsyncClient.java:189) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) >> attempt_201504280858_0001_17_0: at >> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) >> attempt_201504280858_0001_17_0: at >> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) >> attempt_201504280858_0001_17_0: Caused by: >> io.netty.channel.ChannelException: failed to open a new selector >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87) >> attempt_201504280858_0001_17_0: at >> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64) >> attempt_201504280858_0001_17_0: ... 24 more >> attempt_201504280858_0001_17_0: Caused by: java.io.IOException: >> Too many open files >> attempt_201504280858_0001_17_0: at sun.nio.ch.IOUtil.makePipe(Native >> Method) >> attempt_201504280858_0001_17_0: at >> sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65) >> attempt_201504280858_0001_17_0: at >> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) >> attempt_201504280858_0001_17_0: at >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126) >> attempt_201504280858_0001_17_0: ... 27 more >> attempt_201504280858_0001_17_0: 15/04/28 08:28:21 INFO >> ipc.AsyncServer: AsyncServer gracefully shutdown >> >> >> -- >> Best Regards, Ed
Re: Bug in Netty-based RPC
Have you checked limit.conf? >From the message it looks like the files opened at underlying system exceed its default limit. On 28 April 2015 at 08:08, Edward J. Yoon wrote: > I tried to run BSP job using netty-based RPC instead of message > bundle, but I received "too many open files". > > -- > > attempt_201504280858_0001_17_0: 15/04/28 08:28:17 INFO > ipc.AsyncClient: AsyncClient startup > attempt_201504280858_0001_17_0: 15/04/28 08:28:21 ERROR > bsp.BSPTask: Error running bsp setup and bsp function. > attempt_201504280858_0001_17_0: java.lang.IllegalStateException: > failed to create a child event loop > attempt_201504280858_0001_17_0: at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68) > attempt_201504280858_0001_17_0: at > io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncClient$Connection.(AsyncClient.java:189) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) > attempt_201504280858_0001_17_0: at > com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) > attempt_201504280858_0001_17_0: at > org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) > attempt_201504280858_0001_17_0: at > org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467) > attempt_201504280858_0001_17_0: at > org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > attempt_201504280858_0001_17_0: at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > attempt_201504280858_0001_17_0: Caused by: > io.netty.channel.ChannelException: failed to open a new selector > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87) > attempt_201504280858_0001_17_0: at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64) > attempt_201504280858_0001_17_0: ... 24 more > attempt_201504280858_0001_17_0: Caused by: java.io.IOException: > Too many open files > attempt_201504280858_0001_17_0: at sun.nio.ch.IOUtil.makePipe(Native > Method) > attempt_201504280858_0001_17_0: at > sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65) > attempt_201504280858_0001_17_0: at > sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) > attempt_201504280858_0001_17_0: at > io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126) > attempt_201504280858_0001_17_0: ... 27 more > attempt_201504280858_0001_17_0: 15/04/28 08:28:21 INFO > ipc.AsyncServer: AsyncServer gracefully shutdown > > > -- > Best Regards, Edward J. Yoon
Bug in Netty-based RPC
I tried to run BSP job using netty-based RPC instead of message bundle, but I received "too many open files". -- attempt_201504280858_0001_17_0: 15/04/28 08:28:17 INFO ipc.AsyncClient: AsyncClient startup attempt_201504280858_0001_17_0: 15/04/28 08:28:21 ERROR bsp.BSPTask: Error running bsp setup and bsp function. attempt_201504280858_0001_17_0: java.lang.IllegalStateException: failed to create a child event loop attempt_201504280858_0001_17_0: at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68) attempt_201504280858_0001_17_0: at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncClient$Connection.(AsyncClient.java:189) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261) attempt_201504280858_0001_17_0: at com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435) attempt_201504280858_0001_17_0: at org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364) attempt_201504280858_0001_17_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467) attempt_201504280858_0001_17_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) attempt_201504280858_0001_17_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) attempt_201504280858_0001_17_0: Caused by: io.netty.channel.ChannelException: failed to open a new selector attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87) attempt_201504280858_0001_17_0: at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64) attempt_201504280858_0001_17_0: ... 24 more attempt_201504280858_0001_17_0: Caused by: java.io.IOException: Too many open files attempt_201504280858_0001_17_0: at sun.nio.ch.IOUtil.makePipe(Native Method) attempt_201504280858_0001_17_0: at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65) attempt_201504280858_0001_17_0: at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) attempt_201504280858_0001_17_0: at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126) attempt_201504280858_0001_17_0: ... 27 more attempt_201504280858_0001_17_0: 15/04/28 08:28:21 INFO ipc.AsyncServer: AsyncServer gracefully shutdown -- Best Regards, Edward J. Yoon
