[
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282032#comment-16282032
]
Dechang Gu commented on DRILL-5967:
-----------------------------------
Tried the fix (gitid f599401) but still hit memory leak:
{code}
2017-12-07 00:30:45,370 ucs-node7.perf.lab
[25d704d2-562a-3cb6-6aa8-0a1e4cd5101f:frag:8:7] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Memory was leaked by query. Memory leaked: (50176)
Allocator(op:8:7:0:HashPartitionSender) 1000000/0/13447168/10000000000
(res/actual/peak/limit)
Fragment 8:7
[Error Id: ed187677-6ef6-460f-84aa-e40148021487 on ucs-node7.perf.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (50176)
Allocator(op:8:7:0:HashPartitionSender) 1000000/0/13447168/10000000000
(res/actual/peak/limit)
Fragment 8:7
[Error Id: ed187677-6ef6-460f-84aa-e40148021487 on ucs-node7.perf.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory
leaked: (50176)
Allocator(op:8:7:0:HashPartitionSender) 1000000/0/13447168/10000000000
(res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:157)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:79)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:414)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:403)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
... 5 common frames omitted
{code}
Besides HashPartitionSender memory leak, some other operators also have memory
leak, e.g.:
{code}
2017-12-07 02:06:20,694 ucs-node7.perf.lab
[25d6ed7c-6119-00d8-1626-cffae53c4b05:frag:11:103] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Memory was leaked by query. Memory leaked: (524288)
Allocator(op:11:103:6:ParquetRowGroupScan) 1000000/524288/10043392/10000000000
(res/actual/peak/limit)
Fragment 11:103
[Error Id: b0b43e46-6867-4d47-b296-0f51b7e982cb on ucs-node7.perf.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (524288)
Allocator(op:11:103:6:ParquetRowGroupScan) 1000000/524288/10043392/10000000000
(res/actual/peak/limit)
Fragment 11:103
[Error Id: b0b43e46-6867-4d47-b296-0f51b7e982cb on ucs-node7.perf.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory
leaked: (524288)
Allocator(op:11:103:6:ParquetRowGroupScan) 1000000/524288/10043392/10000000000
(res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:157)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:79)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:414)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:403)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
... 5 common frames omitted
{code}
> Memory leak by HashPartitionSender
> ----------------------------------
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The error found by [[email protected]] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 1000000/9216/12831744/10000000000
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 1000000/9216/12831744/10000000000
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 1000000/9216/12831744/10000000000
> (res/actual/peak/limit)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
