Thanks Padma, it worked. Regards, *Anup Tiwari*
On Wed, Apr 19, 2017 at 1:13 AM, Kunal Khatua <[email protected]> wrote: > Could you also share the profiles for the failed queries as well? > > > Thanks > > Kunal > > ________________________________ > From: Padma Penumarthy <[email protected]> > Sent: Tuesday, April 18, 2017 7:18:08 AM > To: [email protected] > Cc: [email protected] > Subject: Re: [Drill 1.10.0] : Memory was leaked by query > > Seems like you are running into DRILL-5435<https://issues.apac > he.org/jira/browse/DRILL-5435>. > Try turning off async parquet reader and see if that helps. > alter session set `store.parquet.reader.pagereader.async`=false; > > Thanks, > Padma > > > On Apr 18, 2017, at 6:14 AM, Anup Tiwari <[email protected]<mai > lto:[email protected]>> wrote: > > Hi Team, > > Please find following information : > > *Cluster configuration :* > Number of Nodes : 5 > Cores/Node : 8 > RAM : 32 > > *Variable values :* > planner.width.max_per_node = 5 > planner.width.max_per_query = 30 > planner.memory.max_query_memory_per_node = 4294967296 > > I am getting following error on simple select statement which is coming 6 > times out of 10 times, let me know if i am missing anything: > > *Query :* > select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl > like '%/j_check%' and ajaxResponse like '%success%true%') limit 5; > > *Error :* > > ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) > Allocator(op:1:24:6:ParquetRowGroupScan) > 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) > > > Fragment 1:24 > > [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on > prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo > p-101.bom-prod.aws.games24x7.com>:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) > Allocator(op:1:24:6:ParquetRowGroupScan) > 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) > > > Fragment 1:24 > > [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on > prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo > p-101.bom-prod.aws.games24x7.com>:31010] > at > org.apache.drill.common.exceptions.UserException$Builder. > build(UserException.java:544) > ~[drill-common-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin > alState(FragmentExecutor.java:293) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup > (FragmentExecutor.java:160) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra > gmentExecutor.java:262) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin > gRunnable.java:38) > [drill-common-1.10.0.jar:1.10.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1142) > [na:1.8.0_72] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:617) > [na:1.8.0_72] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (1048576) > Allocator(op:1:24:6:ParquetRowGroupScan) > 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) > > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502) > ~[drill-memory-base-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(Operator > ContextImpl.java:149) > ~[drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(F > ragmentContext.java:422) > ~[drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411) > ~[drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu > tResources(FragmentExecutor.java:318) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup > (FragmentExecutor.java:155) > [drill-java-exec-1.10.0.jar:1.10.0] > ... 5 common frames omitted > 2017-04-18 18:21:54,172 [BitServer-4] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING > --> CANCELLATION_REQUESTED > 2017-04-18 18:21:54,172 [BitServer-4] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report: > CANCELLATION_REQUESTED > 2017-04-18 18:21:54,173 [BitServer-4] WARN > o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist. > 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested > CANCELLATION_REQUESTED --> FAILED > 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED > --> FAILED > 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED > --> FINISHED > 2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21] > ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) > Allocator(op:1:21:6:ParquetRowGroupScan) > 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) > > > Fragment 1:21 > > [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on > prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo > p-101.bom-prod.aws.games24x7.com>:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) > Allocator(op:1:21:6:ParquetRowGroupScan) > 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) > > > Fragment 1:21 > > [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on > prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo > p-101.bom-prod.aws.games24x7.com>:31010] > at > org.apache.drill.common.exceptions.UserException$Builder. > build(UserException.java:544) > ~[drill-common-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin > alState(FragmentExecutor.java:293) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup > (FragmentExecutor.java:160) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra > gmentExecutor.java:262) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin > gRunnable.java:38) > [drill-common-1.10.0.jar:1.10.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1142) > [na:1.8.0_72] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:617) > [na:1.8.0_72] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (1048576) > Allocator(op:1:21:6:ParquetRowGroupScan) > 1000000/1048576/27140096/10000000000 (res/actual/peak/limit) > > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502) > ~[drill-memory-base-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(Operator > ContextImpl.java:149) > ~[drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(F > ragmentContext.java:422) > ~[drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411) > ~[drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu > tResources(FragmentExecutor.java:318) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup > (FragmentExecutor.java:155) > [drill-java-exec-1.10.0.jar:1.10.0] > ... 5 common frames omitted > > > Regards, > *Anup Tiwari* > >
