Hello Anurag Mantripragada, Vihang Karajgaonkar, Todd Lipcon, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15985 to look at the new patch set (#10). Change subject: IMPALA-9778: Refactor HdfsPartition to be immutable ...................................................................... IMPALA-9778: Refactor HdfsPartition to be immutable Making HdfsPartition immutable has several benefits: - HdfsPartition can be shared across table versions. In full catalog update mode, catalog update can ignore unchanged partitions (IMPALA-3234) and send the update in partition granularity. - Aborted DDL/DMLs won't leave partition metadata in a bad shape, which usually requires invalidation to recover. - Fetch-on-demand coordinators can cache partition meta using the partition id as the key. When table version updates, only metadata of changed partitions needs to be reloaded (IMPALA-7533). - In the work of decoupling partitions from tables (IMPALA-3127), we don't need to assign a catalog version to partitions since the partition ids already identify the partitions. Changes: - Make all fields of HdfsPartition immutable (final). Move HdfsPartition constructor logics and all its update methods into HdfsPartition.Builder. - Refactor in-place updates on HdfsPartition to be creating a new one and dropping the old one. HdfsPartition.Builder represents the in-progress modifications. Once all modifications done, call its build() method to create the new HdfsPartition instance. The old HdfsPartition instance is only replaced at the end of the modifications. - Move the "dirty" marker of HdfsPartition into a map of HdfsTable. It maps from the old partition id to the in-progress partition builder. For these "dirty" partitions, we'll reload its HMS meta and file meta. Tests: - No new tests are added since the existing tests have provided sufficient coverage - Run CORE tests Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c --- M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 10 files changed, 756 insertions(+), 490 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/15985/10 -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>