[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441794#comment-16441794 ] Robert Hou commented on DRILL-5900: --- I looked at the last few runs, and it looks like the test is consistently passing. I'm going to close it. > Regression: TPCH query encounters random IllegalStateException: Memory was > leaked by query > -- > > Key: DRILL-5900 > URL: https://issues.apache.org/jira/browse/DRILL-5900 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Robert Hou >Assignee: Timothy Farkas >Priority: Blocker > Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, > drillbit.log.node81, drillbit.log.node88 > > > This is a random failure in the TPCH-SF100-baseline run. The test is > /root/drillAutomation/framework-master/framework/resources/Advanced/tpch/tpch_sf1/original/parquet/query17.sql. > This test has passed before. > TPCH query 6: > {noformat} > SELECT > SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY > FROM > lineitem L, > part P > WHERE > P.P_PARTKEY = L.L_PARTKEY > AND P.P_BRAND = 'BRAND#13' > AND P.P_CONTAINER = 'JUMBO CAN' > AND L.L_QUANTITY < ( > SELECT > 0.2 * AVG(L2.L_QUANTITY) > FROM > lineitem L2 > WHERE > L2.L_PARTKEY = P.P_PARTKEY > ) > {noformat} > Error is: > {noformat} > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) > ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > ... 5 common frames omitted > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:6:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:6:0: State to
[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439930#comment-16439930 ] Timothy Farkas commented on DRILL-5900: --- Rebased onto the latest master [~rhou] please try to reproduce when you have time. > Regression: TPCH query encounters random IllegalStateException: Memory was > leaked by query > -- > > Key: DRILL-5900 > URL: https://issues.apache.org/jira/browse/DRILL-5900 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Robert Hou >Assignee: Timothy Farkas >Priority: Blocker > Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, > drillbit.log.node81, drillbit.log.node88 > > > This is a random failure in the TPCH-SF100-baseline run. The test is > /root/drillAutomation/framework-master/framework/resources/Advanced/tpch/tpch_sf1/original/parquet/query17.sql. > This test has passed before. > TPCH query 6: > {noformat} > SELECT > SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY > FROM > lineitem L, > part P > WHERE > P.P_PARTKEY = L.L_PARTKEY > AND P.P_BRAND = 'BRAND#13' > AND P.P_CONTAINER = 'JUMBO CAN' > AND L.L_QUANTITY < ( > SELECT > 0.2 * AVG(L2.L_QUANTITY) > FROM > lineitem L2 > WHERE > L2.L_PARTKEY = P.P_PARTKEY > ) > {noformat} > Error is: > {noformat} > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) > ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > ... 5 common frames omitted > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:6:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:6:0: State to report: FINISHED
[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387034#comment-16387034 ] Timothy Farkas commented on DRILL-5900: --- [~rhou] Can you please try to reproduce the issue? I've added Salim's extra debugging for memory leaks in this branch https://github.com/ilooner/drill/tree/DRILL-5900 . To activate the debugging run Drill with java assertions enabled. > Regression: TPCH query encounters random IllegalStateException: Memory was > leaked by query > -- > > Key: DRILL-5900 > URL: https://issues.apache.org/jira/browse/DRILL-5900 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Robert Hou >Assignee: Timothy Farkas >Priority: Blocker > Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, > drillbit.log.node81, drillbit.log.node88 > > > This is a random failure in the TPCH-SF100-baseline run. The test is > /root/drillAutomation/framework-master/framework/resources/Advanced/tpch/tpch_sf1/original/parquet/query17.sql. > This test has passed before. > TPCH query 6: > {noformat} > SELECT > SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY > FROM > lineitem L, > part P > WHERE > P.P_PARTKEY = L.L_PARTKEY > AND P.P_BRAND = 'BRAND#13' > AND P.P_CONTAINER = 'JUMBO CAN' > AND L.L_QUANTITY < ( > SELECT > 0.2 * AVG(L2.L_QUANTITY) > FROM > lineitem L2 > WHERE > L2.L_PARTKEY = P.P_PARTKEY > ) > {noformat} > Error is: > {noformat} > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) > ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > ... 5 common frames omitted > 2017-10-23 10:34:55,989
[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371851#comment-16371851 ] Timothy Farkas commented on DRILL-5900: --- I believe Salim has fixed a memory leak in parquet row group scan. I will sync up with him to see if this is the same issue. > Regression: TPCH query encounters random IllegalStateException: Memory was > leaked by query > -- > > Key: DRILL-5900 > URL: https://issues.apache.org/jira/browse/DRILL-5900 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Robert Hou >Assignee: Timothy Farkas >Priority: Blocker > Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, > drillbit.log.node81, drillbit.log.node88 > > > This is a random failure in the TPCH-SF100-baseline run. The test is > /root/drillAutomation/framework-master/framework/resources/Advanced/tpch/tpch_sf1/original/parquet/query17.sql. > This test has passed before. > TPCH query 6: > {noformat} > SELECT > SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY > FROM > lineitem L, > part P > WHERE > P.P_PARTKEY = L.L_PARTKEY > AND P.P_BRAND = 'BRAND#13' > AND P.P_CONTAINER = 'JUMBO CAN' > AND L.L_QUANTITY < ( > SELECT > 0.2 * AVG(L2.L_QUANTITY) > FROM > lineitem L2 > WHERE > L2.L_PARTKEY = P.P_PARTKEY > ) > {noformat} > Error is: > {noformat} > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) > ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > ... 5 common frames omitted > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:6:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - >
[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216181#comment-16216181 ] Paul Rogers commented on DRILL-5900: The key line is this one: {noformat} Memory leaked: (2097152) Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 (res/actual/peak/limit) {noformat} This is a single buffer of size 0x20. (Or, two buffers half that size, etc.) Does Parquet allocate a working buffer of this size that it fails to release under some situations? > Regression: TPCH query encounters random IllegalStateException: Memory was > leaked by query > -- > > Key: DRILL-5900 > URL: https://issues.apache.org/jira/browse/DRILL-5900 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Robert Hou >Assignee: Pritesh Maker >Priority: Blocker > Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, > drillbit.log.node81, drillbit.log.node88 > > > This is a random failure. This test has passed before. > TPCH query 6: > {noformat} > SELECT > SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY > FROM > lineitem L, > part P > WHERE > P.P_PARTKEY = L.L_PARTKEY > AND P.P_BRAND = 'BRAND#13' > AND P.P_CONTAINER = 'JUMBO CAN' > AND L.L_QUANTITY < ( > SELECT > 0.2 * AVG(L2.L_QUANTITY) > FROM > lineitem L2 > WHERE > L2.L_PARTKEY = P.P_PARTKEY > ) > {noformat} > Error is: > {noformat} > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) > ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > ... 5 common frames omitted > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:6:0] INFO >
[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216035#comment-16216035 ] Robert Hou commented on DRILL-5900: --- The foreman is in node 81. Node 88 encounters the memory error. > Regression: TPCH query encounters random IllegalStateException: Memory was > leaked by query > -- > > Key: DRILL-5900 > URL: https://issues.apache.org/jira/browse/DRILL-5900 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Robert Hou >Assignee: Pritesh Maker >Priority: Blocker > Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, > drillbit.log.node81, drillbit.log.node88 > > > This is a random failure. This test has passed before. > TPCH query 6: > {noformat} > SELECT > SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY > FROM > lineitem L, > part P > WHERE > P.P_PARTKEY = L.L_PARTKEY > AND P.P_BRAND = 'BRAND#13' > AND P.P_CONTAINER = 'JUMBO CAN' > AND L.L_QUANTITY < ( > SELECT > 0.2 * AVG(L2.L_QUANTITY) > FROM > lineitem L2 > WHERE > L2.L_PARTKEY = P.P_PARTKEY > ) > {noformat} > Error is: > {noformat} > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > Fragment 8:2 > [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) > ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (2097152) > Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155) > [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] > ... 5 common frames omitted > 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:6:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:6:0: State to report: FINISHED > {noformat} > sys.version is: > 1.12.0-SNAPSHOT b0c4e0486d6d4620b04a1bb8198e959d433b4840 > DRILL-5876: Use openssl profile to include netty-tcnative dependency with the >