Repository: carbondata Updated Branches: refs/heads/master b716865bf -> 877efe1a4
[CARBONDATA-3122]CarbonReader memory leak Issue Detail CarbonReader has List of initialized RecordReader for each Split and each split holds page data till the reference of RecordReader is present in the List . Same is applicable for GC once user comes out from his/her calling method ( not cleaned even in close() ) but till then from each split , last page will be in memory which is not correct. For ex. if 1K carbon files then last page ( ~32K * 100 ,size if 100 String columns in memory ) of each file will be in memory till last split so total ~3GB memory will be occupied ( 1K * 32K * 100 . Check heap dump of 3 split after reader.close() is called ,It is be seen that currentreader+all list reader are still holding memory Solution Once reader is finished assign currentReader to null in RecordReader List. OR Assign future object as null in org.apache.carbondata.core.scan.processor.DataBlockIterator#close() Solution 2 is adopted so that it will give benefit to other than CarbonReader Flow. This closes #2944 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/877efe1a Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/877efe1a Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/877efe1a Branch: refs/heads/master Commit: 877efe1a44607aa12edb66f58445db08fd3ca63d Parents: b716865 Author: BJangir <[email protected]> Authored: Thu Nov 22 22:34:32 2018 +0530 Committer: ravipesala <[email protected]> Committed: Tue Nov 27 12:14:12 2018 +0530 ---------------------------------------------------------------------- .../apache/carbondata/core/scan/processor/DataBlockIterator.java | 1 + .../main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java | 1 + 2 files changed, 2 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/carbondata/blob/877efe1a/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java ---------------------------------------------------------------------- diff --git a/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java b/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java index 1eeb579..4f635c6 100644 --- a/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java +++ b/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java @@ -262,6 +262,7 @@ public class DataBlockIterator extends CarbonIterator<List<Object[]>> { if (blockletScannedResult != null) { blockletScannedResult.freeMemory(); } + future = null; } catch (InterruptedException | ExecutionException e) { throw new RuntimeException(e); } http://git-wip-us.apache.org/repos/asf/carbondata/blob/877efe1a/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java ---------------------------------------------------------------------- diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java index eca3396..1dfead3 100644 --- a/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java +++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java @@ -166,6 +166,7 @@ public class CarbonRecordReader<T> extends AbstractRecordReader<T> { } // close read support readSupport.close(); + carbonIterator.close(); try { queryExecutor.finish(); } catch (QueryExecutionException e) {
