Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15985 )
Change subject: IMPALA-9778: Refactor partition modifications in DDL/DMLs ...................................................................... IMPALA-9778: Refactor partition modifications in DDL/DMLs After this patch, in DDL/DMLs that update metadata of partitions, instead of updating partitions in place, we always create new ones and use them to replace the existing instances. This is guarded by making HdfsPartition immutable. There are several benefits for this: - HdfsPartition can be shared across table versions. In full catalog update mode, catalog update can ignore unchanged partitions (IMPALA-3234) and send the update in partition granularity. - Aborted DDL/DMLs won't leave partition metadata in a bad shape (e.g. IMPALA-8406), which usually requires invalidation to recover. - Fetch-on-demand coordinators can cache partition meta using the partition id as the key. When table version updates, only metadata of changed partitions need to be reloaded (IMPALA-7533). - In the work of decoupling partitions from tables (IMPALA-3127), we don't need to assign a catalog version to partitions since the partition ids already identify the partitions. However, HdfsPartition is not strictly immutable. Although all its fields are final, some fields are still referencing mutable objects. We need more refactoring to achieve this. This patch focuses on refactoring the DDL/DML code paths. Changes: - Make all fields of HdfsPartition final. Move HdfsPartition constructor logics and all its update methods into HdfsPartition.Builder. - Refactor in-place updates on HdfsPartition to be creating a new one and dropping the old one. HdfsPartition.Builder represents the in-progress modifications. Once all modifications are done, call its build() method to create the new HdfsPartition instance. The old HdfsPartition instance is only replaced at the end of the modifications. - Move the "dirty" marker of HdfsPartition into a map of HdfsTable. It maps from the old partition id to the in-progress partition builder. For "dirty" partitions, we’ll reload its HMS meta and file meta. Tests: - No new tests are added since the existing tests already provide sufficient coverage - Run CORE tests Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Reviewed-on: http://gerrit.cloudera.org:8080/15985 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 11 files changed, 819 insertions(+), 512 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c Gerrit-Change-Number: 15985 Gerrit-PatchSet: 17 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Anurag Mantripragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
