Could you also share the profiles for the failed queries as well?
Thanks Kunal ________________________________ From: Padma Penumarthy <ppenumar...@mapr.com> Sent: Tuesday, April 18, 2017 7:18:08 AM To: user@drill.apache.org Cc: d...@drill.apache.org Subject: Re: [Drill 1.10.0] : Memory was leaked by query Seems like you are running into DRILL-5435<https://issues.apache.org/jira/browse/DRILL-5435>. Try turning off async parquet reader and see if that helps. alter session set `store.parquet.reader.pagereader.async`=false; Thanks, Padma On Apr 18, 2017, at 6:14 AM, Anup Tiwari <anup.tiw...@games24x7.com<mailto:anup.tiw...@games24x7.com>> wrote: Hi Team, Please find following information : *Cluster configuration :* Number of Nodes : 5 Cores/Node : 8 RAM : 32 *Variable values :* planner.width.max_per_node = 5 planner.width.max_per_query = 30 planner.memory.max_query_memory_per_node = 4294967296 I am getting following error on simple select statement which is coming 6 times out of 10 times, let me know if i am missing anything: *Query :* select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl like '%/j_check%' and ajaxResponse like '%success%true%') limit 5; *Error :* ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) Allocator(op:1:24:6:ParquetRowGroupScan) 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) Fragment 1:24 [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) Allocator(op:1:24:6:ParquetRowGroupScan) 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) Fragment 1:24 [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.10.0.jar:1.10.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) Allocator(op:1:24:6:ParquetRowGroupScan) 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502) ~[drill-memory-base-1.10.0.jar:1.10.0] at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149) ~[drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422) ~[drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411) ~[drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) [drill-java-exec-1.10.0.jar:1.10.0] ... 5 common frames omitted 2017-04-18 18:21:54,172 [BitServer-4] INFO o.a.d.e.w.fragment.FragmentExecutor - 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING --> CANCELLATION_REQUESTED 2017-04-18 18:21:54,172 [BitServer-4] INFO o.a.d.e.w.f.FragmentStatusReporter - 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report: CANCELLATION_REQUESTED 2017-04-18 18:21:54,173 [BitServer-4] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. 2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist. 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] INFO o.a.d.e.w.fragment.FragmentExecutor - 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested CANCELLATION_REQUESTED --> FAILED 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] INFO o.a.d.e.w.fragment.FragmentExecutor - 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED --> FAILED 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] INFO o.a.d.e.w.fragment.FragmentExecutor - 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED --> FINISHED 2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) Allocator(op:1:21:6:ParquetRowGroupScan) 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) Fragment 1:21 [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) Allocator(op:1:21:6:ParquetRowGroupScan) 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) Fragment 1:21 [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.10.0.jar:1.10.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) Allocator(op:1:21:6:ParquetRowGroupScan) 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502) ~[drill-memory-base-1.10.0.jar:1.10.0] at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149) ~[drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422) ~[drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411) ~[drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318) [drill-java-exec-1.10.0.jar:1.10.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) [drill-java-exec-1.10.0.jar:1.10.0] ... 5 common frames omitted Regards, *Anup Tiwari*
