[
https://issues.apache.org/jira/browse/TAJO-675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keuntae Park updated TAJO-675:
------------------------------
Summary: size of freamDecoder should be increase (was: maxContentLength of
HttpChunkAggregator() should be increases)
> size of freamDecoder should be increase
> ---------------------------------------
>
> Key: TAJO-675
> URL: https://issues.apache.org/jira/browse/TAJO-675
> Project: Tajo
> Issue Type: Improvement
> Reporter: Keuntae Park
>
> As Repartitioner::scheduleHashShuffledFetches re-optimizes the number of
> tasks, sometimes one task may fetch results of thousands of tasks.
> In my case, there is a case that one task fetches from 9240 tasks,
> and got following exception
> {noformat}
> 2014-03-07 13:29:24,240 ERROR rpc.AsyncRpcClient
> (AsyncRpcClient.java:exceptionCaught(220)) - skt-rf-01/50.1.103.1:28093,class
> org.apache.tajo.ipc.QueryMasterProtocol,Adjusted frame length exceeds
> 2097152: 7729393 - discarded
> org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame
> length exceeds 2097152: 7729393 - discarded
> at
> org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:417)
> at
> org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:405)
> at
> org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:320)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2014-03-07 13:29:24,242 ERROR rpc.AsyncRpcClient
> (AsyncRpcClient.java:exceptionCaught(235)) - RPC
> Exception:org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted
> frame length exceeds 2097152: 7729393 - discarded
> {noformat}
> The reason seems that frameDecoder setting in getPipeline() of
> ProtoPipelineFactory.java is 2MB
> {noformat}
> public ChannelPipeline getPipeline() throws Exception {
> ChannelPipeline p = Channels.pipeline();
> p.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(1048576*2, 0,
> 4,
> 0, 4));
> p.addLast("protobufDecoder", new ProtobufDecoder(defaultInstance));
> p.addLast("frameEncoder", new LengthFieldPrepender(4));
> p.addLast("protobufEncoder", new ProtobufEncoder());
> p.addLast("handler", handler);
> return p;
> }
> {noformat}
> I think temporal solution for the issue is just increasing the size.
> However, I'm not sure what the proper size is.
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
