zhangqianqiong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21922
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`. ...................................................................... IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`. After applying the merge request 'IMPALA-10502: Handle CREATE/DROP events correctly', the `alterTableRecoverPartitions` method changed from batching the `add_partitions` calls to invoking `addHmsPartitions` all at once. However, for tables with a huge number of partitions, this can result in the creation of a huge temporary object, List<Partitions>, leading to OutOfMemory. In my test environment, where the catalogd JVM Xmx was set to 2GB, running the end-to-end test custom_cluster/test_wide_table_operations.py on a table with 2000 columns and 50,000 partitions during the recover partitions operation caused catalogd to run into a 'Java heap space' OutOfMemoryError. An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions), which overwhelmed memory resources. To resolve this issue, we propose batching the addHmsPartitions calls, ensuring that temporary objects are released after each batch operation. This solution was tested and verified to resolve the OutOfMemoryError, ensuring system stability when handling a large number of partitions. Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 16 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/21922/1 -- To view, visit http://gerrit.cloudera.org:8080/21922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592 Gerrit-Change-Number: 21922 Gerrit-PatchSet: 1 Gerrit-Owner: zhangqianqiong <[email protected]>
