Hi Team,
Please find following information :
*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32
*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296
I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:
*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
*Error :*
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*
