[jira] [Updated] (DRILL-5718) java.lang.IllegalStateException: Memory was leaked by query

2017-08-13 Thread Muhammad Gelbana (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Muhammad Gelbana updated DRILL-5718:

Environment: 
uname -a
Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

48 Cores
Assigned to Drill 25 GB heap and 200 GB direct memory
The machine has a total of 500GB of RAM, 250 of it is assigned to another 
application. That application and Drill are the only considerable applications 
on that machine.

  was:
uname -a
Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

48 Cores
Assigned to Drill 25 GB heap and 200 GB direct memory


> java.lang.IllegalStateException: Memory was leaked by query
> ---
>
> Key: DRILL-5718
> URL: https://issues.apache.org/jira/browse/DRILL-5718
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Execution - RPC
>Affects Versions: 1.9.0, 1.11.0
> Environment: uname -a
> Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 
> 2017 x86_64 x86_64 x86_64 GNU/Linux
> 48 Cores
> Assigned to Drill 25 GB heap and 200 GB direct memory
> The machine has a total of 500GB of RAM, 250 of it is assigned to another 
> application. That application and Drill are the only considerable 
> applications on that machine.
>Reporter: Muhammad Gelbana
> Attachments: drillbit.out.tar.gz
>
>
> Configurations
> {noformat}
> planner.memory.max_query_memory_per_node: 17179869184 (16 GB)
> planner.width.max_per_node: 48
> store.parquet.block-size: 134217728 (128 MB, this is the block size used to 
> create the parquet files)
> {noformat}
> {noformat}
> Fragment 0:0
> [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
> [BitServer-4] INFO org.apache.drill.exec.work.fragment.FragmentExecutor - 
> 267104f2-e48d-1d66-63f4-387848c1ccf2:1:10: State change requested RUNNING --> 
> CANCELLATION_REQUESTED
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> ChannelClosedException: Channel closed /127.0.0.1:31010 <--> /127.0.0.1:40404.
> Fragment 0:0
> [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed 
> /127.0.0.1:31010 <--> /127.0.0.1:40404.
> at 
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164)
> at 
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at 
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at 
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at 
> io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
> at 
> io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
> at 
> io.netty.channel.AbstractChannelHa

[jira] [Updated] (DRILL-5718) java.lang.IllegalStateException: Memory was leaked by query

2017-08-12 Thread Muhammad Gelbana (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Muhammad Gelbana updated DRILL-5718:

Environment: 
uname -a
Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

48 Cores
Assigned to Drill 25 GB heap and 200 GB direct memory

  was:
Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

48 Cores
25 GB Heap
200 GB Direct memory


> java.lang.IllegalStateException: Memory was leaked by query
> ---
>
> Key: DRILL-5718
> URL: https://issues.apache.org/jira/browse/DRILL-5718
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Execution - RPC
>Affects Versions: 1.9.0, 1.11.0
> Environment: uname -a
> Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 
> 2017 x86_64 x86_64 x86_64 GNU/Linux
> 48 Cores
> Assigned to Drill 25 GB heap and 200 GB direct memory
>Reporter: Muhammad Gelbana
> Attachments: drillbit.out.tar.gz
>
>
> Configurations
> {noformat}
> planner.memory.max_query_memory_per_node: 17179869184 (16 GB)
> planner.width.max_per_node: 48
> store.parquet.block-size: 134217728 (128 MB, this is the block size used to 
> create the parquet files)
> {noformat}
> {noformat}
> Fragment 0:0
> [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
> [BitServer-4] INFO org.apache.drill.exec.work.fragment.FragmentExecutor - 
> 267104f2-e48d-1d66-63f4-387848c1ccf2:1:10: State change requested RUNNING --> 
> CANCELLATION_REQUESTED
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> ChannelClosedException: Channel closed /127.0.0.1:31010 <--> /127.0.0.1:40404.
> Fragment 0:0
> [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed 
> /127.0.0.1:31010 <--> /127.0.0.1:40404.
> at 
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164)
> at 
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at 
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at 
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at 
> io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
> at 
> io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466)
> at 
> io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:187)
> at 
> org.apache.drill.exec.rpc.BasicServer$LoggingReadTimeoutHandler.readTimedOut(BasicServer.java:122)
> at 
> io.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask.run(ReadTimeoutHandler.java:2

[jira] [Updated] (DRILL-5718) java.lang.IllegalStateException: Memory was leaked by query

2017-08-12 Thread Muhammad Gelbana (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Muhammad Gelbana updated DRILL-5718:

Attachment: drillbit.out.tar.gz

Full logs

> java.lang.IllegalStateException: Memory was leaked by query
> ---
>
> Key: DRILL-5718
> URL: https://issues.apache.org/jira/browse/DRILL-5718
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Execution - RPC
>Affects Versions: 1.9.0, 1.11.0
> Environment: Linux iWebGelbanaDev 2.6.32-696.1.1.el6.x86_64 #1 SMP 
> Tue Apr 11 17:13:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> 48 Cores
> 25 GB Heap
> 200 GB Direct memory
>Reporter: Muhammad Gelbana
> Attachments: drillbit.out.tar.gz
>
>
> Configurations
> {noformat}
> planner.memory.max_query_memory_per_node: 17179869184 (16 GB)
> planner.width.max_per_node: 48
> store.parquet.block-size: 134217728 (128 MB, this is the block size used to 
> create the parquet files)
> {noformat}
> {noformat}
> Fragment 0:0
> [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
> [BitServer-4] INFO org.apache.drill.exec.work.fragment.FragmentExecutor - 
> 267104f2-e48d-1d66-63f4-387848c1ccf2:1:10: State change requested RUNNING --> 
> CANCELLATION_REQUESTED
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> ChannelClosedException: Channel closed /127.0.0.1:31010 <--> /127.0.0.1:40404.
> Fragment 0:0
> [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed 
> /127.0.0.1:31010 <--> /127.0.0.1:40404.
> at 
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164)
> at 
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at 
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at 
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at 
> io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
> at 
> io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
> at 
> io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466)
> at 
> io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:187)
> at 
> org.apache.drill.exec.rpc.BasicServer$LoggingReadTimeoutHandler.readTimedOut(BasicServer.java:122)
> at 
> io.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask.run(ReadTimeoutHandler.java:212)
> at 
> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
> at 
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> at io.netty.channel.nio.NioEventLoop.