zhangqianqiong has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`. ...................................................................... Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG@20 PS2, Line 20: An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object : contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions), : which overwhelmed memory resources. > This seems to be resolved by IMPALA-11812. Do you have that patch in your d I have reproduced the issue on the latest master branch. Although IMPALA-11812 provides a partial fix, it doesn't fully resolve the problem. The root cause lies in the alterTableRecoverPartitions method, specifically in the line List<Partition> addedPartitions = addHmsPartitions(msClient, tbl, hmsPartitions, partitionToEventId, true, catalogTimeline);. The size of the returned result object is too large, leading to OutOfMemory. -- To view, visit http://gerrit.cloudera.org:8080/21922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592 Gerrit-Change-Number: 21922 Gerrit-PatchSet: 2 Gerrit-Owner: zhangqianqiong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: zhangqianqiong <[email protected]> Gerrit-Comment-Date: Mon, 14 Oct 2024 02:55:32 +0000 Gerrit-HasComments: Yes
