Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22538
Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread ...................................................................... IMPALA-13783: Fix huge temp list created in catalog-update thread In catalogd, the catalog-update thread collects new/deleted catalog objects since the last update. When collecting new partitions of a HdfsTable, a list of TCatalogObject is created. Each TCatalogObject has a THdfsPartition corresponding to a new partition. See HdfsTable#getNewPartitionsSinceLastUpdate(). This temp list could be huge and occupying a comparable memory footprint to the table metadata. E.g. for a HdfsTable that has 6M partitions and 6M files (one file per partition), the HdfsTable object itself takes 7.3GB in memory. This temp list takes 8.9GB. See more details in the JIRA description. This patch fixes the issue by iterating the partition map directly to avoid creating this temp list. For each partition, we collect its thrift representation and add it into the topic update directly. In local-catalog mode, the thrift representation of the partition can be simplified to just contain the names of db, table, partition and the partition ids. This is also optimized to reduce the topic update size in local-catalog mode. Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d --- M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 4 files changed, 28 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/22538/1 -- To view, visit http://gerrit.cloudera.org:8080/22538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d Gerrit-Change-Number: 22538 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
