[
https://issues.apache.org/jira/browse/DRILL-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250135#comment-15250135
]
ASF GitHub Bot commented on DRILL-3317:
---------------------------------------
Github user adeneche commented on the pull request:
https://github.com/apache/drill/pull/446#issuecomment-212488188
So I spent more time looking at ways to create a unit test for this, and I
don't think it's possible:
First, current implementation of ProtobufLengthDecoder for the data tunnel
uses the root allocator, which is weird and changing the rpc max allocation
won't help in this case.
Second, even after I changed the implementation and made sure
ProtobufLengthDecoder uses the rpc allocator, this same allocator is used by
netty too and you can't control where it will run out of memory, in most of my
experiments it run out of memory in ByteToMessageDecoder, so writing a unit
test using this won't be reliable as it won't consistently reproduce the issue.
> when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is
> just logged and nothing else is done
> ---------------------------------------------------------------------------------------------------------------
>
> Key: DRILL-3317
> URL: https://issues.apache.org/jira/browse/DRILL-3317
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC
> Reporter: Deneche A. Hakim
> Assignee: Jacques Nadeau
> Fix For: 1.7.0
>
>
> Trying to reproduce DRILL-3241 I sometimes get the following error in the
> logs:
> {noformat}
> ERROR: Out of memory outside any particular fragment.
> at
> org.apache.drill.exec.rpc.data.DataResponseHandlerImpl.informOutOfMemory(DataResponseHandlerImpl.java:40)
> at
> org.apache.drill.exec.rpc.data.DataServer$2.handle(DataServer.java:227)
> at
> org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:87)
> at
> org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> WARN: Failure allocating buffer on incoming stream due to memory limits.
> Current Allocation: 1372678764.
> at
> org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:85)
> at
> org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> {noformat}
> ProtobufLengthDecoder.decode() does call OutOfMemoryHandler.handle() which
> calls DataResponseHandlerImpl.informOutOfMemory() which just logs the error
> in the logs.
> If we have fragments waiting for data they will be stuck waiting forever, and
> the query will hang (behavior observed in DRILL-3241
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
