Repository: carbondata Updated Branches: refs/heads/master 0e8588744 -> 1be990f66
[CARBONDATA-3026] clear expired property that may cause GC problem During data loading, we will write some temp files (sort temp files and temp fact data files) in some locations. In currently implementation, we will add the locations to the CarbonProperties and associated it with a special key that refers to the data loading. After data loading, the temp locations are cleared, but the added property is still remain in the CarbonProperties and never to be cleared. This will cause the CarbonProperties object growing bigger and bigger and lead to OOM problems if the thrift-server is a long time running service. A local test shows that after adding different properties for 11 Billion times, the OOM happens. In this commit, I clear the property for the locations when we clear the locations. This closes #2833 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/1be990f6 Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/1be990f6 Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/1be990f6 Branch: refs/heads/master Commit: 1be990f66296174a34044d67c79cffae37f06e86 Parents: 0e85887 Author: xuchuanyin <[email protected]> Authored: Thu Oct 18 14:12:30 2018 +0800 Committer: Jacky Li <[email protected]> Committed: Thu Oct 18 20:49:44 2018 +0800 ---------------------------------------------------------------------- .../org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java | 3 --- .../carbondata/processing/loading/TableProcessingOperations.java | 1 + 2 files changed, 1 insertion(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/carbondata/blob/1be990f6/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java ---------------------------------------------------------------------- diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java index f0ad94d..f0f2858 100644 --- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java +++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java @@ -267,9 +267,6 @@ public class CarbonTableOutputFormat extends FileOutputFormat<NullWritable, Obje iterator.closeWriter(true); } dataLoadExecutor.close(); - // clean up the folders and files created locally for data load operation - TableProcessingOperations.deleteLocalDataLoadFolderLocation(loadModel, false, false); - throw new RuntimeException(e); } finally { ThreadLocalSessionInfo.unsetAll(); http://git-wip-us.apache.org/repos/asf/carbondata/blob/1be990f6/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java ---------------------------------------------------------------------- diff --git a/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java b/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java index 89da224..f08de59 100644 --- a/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java +++ b/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java @@ -146,6 +146,7 @@ public class TableProcessingOperations { } }); } finally { + CarbonProperties.getInstance().removeProperty(tempLocationKey); if (null != localFolderDeletionService) { localFolderDeletionService.shutdown(); }
