[
https://issues.apache.org/jira/browse/DRILL-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213368#comment-15213368
]
ASF GitHub Bot commented on DRILL-3317:
---------------------------------------
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/446#discussion_r57524389
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/ProtobufLengthDecoder.java ---
@@ -82,15 +79,7 @@ protected void decode(ChannelHandlerContext ctx, ByteBuf
in, List<Object> out) t
} else {
// need to make buffer copy, otherwise netty will try to refill
this buffer if we move the readerIndex forward...
// TODO: Can we avoid this copy?
- ByteBuf outBuf;
- try {
- outBuf = allocator.buffer(length);
- } catch (OutOfMemoryException e) {
- logger.warn("Failure allocating buffer on incoming stream due
to memory limits. Current Allocation: {}.", allocator.getAllocatedMemory());
- in.resetReaderIndex();
- outOfMemoryHandler.handle();
- return;
- }
+ ByteBuf outBuf = allocator.buffer(length);
--- End diff --
So I was looking at how we are using `ProtobufLengthDecoder` in our code,
and we use in the control, server and user tunnels both on the client and
server side.
If I am not mistaken, the only derived class from ProtobufLengthDecoder
that actually uses an rpc allocator is UserProtobufLengthDecoder and only on
the server side of the user tunnel. All other implementations use the root
allocator.
This may actually explain why I wasn't able to reproduce this issue no
matter how much I reduced the "rpc:bit-data" max allocation.
am I missing something ?
> when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is
> just logged and nothing else is done
> ---------------------------------------------------------------------------------------------------------------
>
> Key: DRILL-3317
> URL: https://issues.apache.org/jira/browse/DRILL-3317
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC
> Reporter: Deneche A. Hakim
> Assignee: Jacques Nadeau
> Fix For: 1.7.0
>
>
> Trying to reproduce DRILL-3241 I sometimes get the following error in the
> logs:
> {noformat}
> ERROR: Out of memory outside any particular fragment.
> at
> org.apache.drill.exec.rpc.data.DataResponseHandlerImpl.informOutOfMemory(DataResponseHandlerImpl.java:40)
> at
> org.apache.drill.exec.rpc.data.DataServer$2.handle(DataServer.java:227)
> at
> org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:87)
> at
> org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> WARN: Failure allocating buffer on incoming stream due to memory limits.
> Current Allocation: 1372678764.
> at
> org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:85)
> at
> org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> {noformat}
> ProtobufLengthDecoder.decode() does call OutOfMemoryHandler.handle() which
> calls DataResponseHandlerImpl.informOutOfMemory() which just logs the error
> in the logs.
> If we have fragments waiting for data they will be stuck waiting forever, and
> the query will hang (behavior observed in DRILL-3241
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
