[
https://issues.apache.org/jira/browse/DRILL-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16407091#comment-16407091
]
Khurram Faraaz commented on DRILL-6229:
---------------------------------------
[~ppenumarthy] we have a test to reproduce the memory leak out of the test
framework. I have attached the test program here, I can share the data file
when you debug the problem.
Run the test program with the default setting of output batch size option and
we will hit the memory leak.
Stack trace from the prompt where the concurrency program was executed.
{noformat}
14:02:05.551 [Client-1] INFO o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is
closed, discarding remaining 191167 byte(s) in buffer.
14:02:05.552 [pool-5-thread-2] WARN o.apache.drill.jdbc.DrillConnection -
Exception while closing client.
java.lang.RuntimeException: Exception while closing
at
org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.close(DrillClient.java:475)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
[drill-jdbc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823)
[drill-jdbc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
[drill-jdbc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205)
[avatica-1.10.0.jar:1.10.0]
at ConcurrentQuery.executeQuery(ConcurrentQuery.java:84) [concur/:na]
at ConcurrentQuery.selectData(ConcurrentQuery.java:58) [concur/:na]
at ConcurrentQuery.run(ConcurrentQuery.java:32) [concur/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory
leaked: (4194304)
Allocator(ROOT) 0/0/39845888/28631367680 (res/actual/peak/limit)
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
~[drill-memory-base-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
... 13 common frames omitted{noformat}
Query used was :
{noformat}
[root@qa102-45 concur]# cat testSQL.q
select flatten(array_of_ints) from tbl_all_types_jsn_to_parquet{noformat}
> memory leak when output_batch_size is set to 4096
> -------------------------------------------------
>
> Key: DRILL-6229
> URL: https://issues.apache.org/jira/browse/DRILL-6229
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.12.0
> Reporter: Khurram Faraaz
> Assignee: Padma Penumarthy
> Priority: Major
>
> When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we
> continuously see in each test run, the memory leak when tests are run from
> test framework.
> Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01
> alter system set `drill.exec.memory.operator.output_batch_size` = 4096;
> To run flatten tests from drill test framework
> bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40
> The below two tests time out and we see the memory leak
> Timeouts:
> /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter24.q
> Timeouts:
> /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter19.q
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR:
> Connection ... (user client) closed unexpectedly. Drillbit down?
> Stack trace printed in drill test framework log
> {noformat}
> [Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ]
> at
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> at
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553)
> at
> oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
> at
> oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
> at
> oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607)
> at
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
> at
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> at java.lang.Thread.run(Thread.java:748)
> Channel is closed, discarding remaining 908 byte(s) in buffer.
> Exception while closing client.
> java.lang.RuntimeException: Exception while closing
> at
> oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
> at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823)
> at
> org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
> at
> oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205)
> at
> org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102)
> at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562)
> at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100)
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (4198400)
> Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit)
> at
> oadd.org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
> at
> oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
