Quanlong Huang has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/15985 )
Change subject: IMPALA-9778: Refactor HdfsPartition to be immutable ...................................................................... IMPALA-9778: Refactor HdfsPartition to be immutable Making HdfsPartition immutable has several benefits: - HdfsPartition can be shared across table versions. In full catalog update mode, catalog update can ignore unchanged partitions (IMPALA-3234) and send the update in partition granularity. - Aborted DDL/DMLs won’t leave partition metadata in a bad shape, which usually requires invalidation to recover. - Fetch-on-demand coordinators can cache partition meta using the partition id as the key. When table version updates, only metadata of changed partitions needs to be reloaded (IMPALA-7533). - In the work of decoupling partitions from tables (IMPALA-3127), we don't need to assign a catalog version to partitions since the partition ids already identify the partitions. Changes: - Make all fields of HdfsPartition immutable (final). Move HdfsPartition constructor logics and all its update methods into HdfsPartition.Builder. - Move the “dirty” marker of HdfsPartition into a map of HdfsTable. It maps from the old partition id to the in-progress partition builder. - Original code paths that modifying HdfsPartition are changed to modify and pass through the HdfsPartition.Builder. The HdfsPartition is only replaced at the end of the modifications. Tests: - Run CORE tests Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 11 files changed, 754 insertions(+), 491 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/15985/9 -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang <[email protected]>
