Hello Anurag Mantripragada, Vihang Karajgaonkar, Todd Lipcon, Tim Armstrong,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15985
to look at the new patch set (#10).
Change subject: IMPALA-9778: Refactor HdfsPartition to be immutable
......................................................................
IMPALA-9778: Refactor HdfsPartition to be immutable
Making HdfsPartition immutable has several benefits:
- HdfsPartition can be shared across table versions. In full catalog
update mode, catalog update can ignore unchanged partitions
(IMPALA-3234) and send the update in partition granularity.
- Aborted DDL/DMLs won't leave partition metadata in a bad shape, which
usually requires invalidation to recover.
- Fetch-on-demand coordinators can cache partition meta using the
partition id as the key. When table version updates, only metadata of
changed partitions needs to be reloaded (IMPALA-7533).
- In the work of decoupling partitions from tables (IMPALA-3127), we
don't need to assign a catalog version to partitions since the
partition ids already identify the partitions.
Changes:
- Make all fields of HdfsPartition immutable (final). Move
HdfsPartition constructor logics and all its update methods into
HdfsPartition.Builder.
- Refactor in-place updates on HdfsPartition to be creating a new one
and dropping the old one. HdfsPartition.Builder represents the
in-progress modifications. Once all modifications done, call its
build() method to create the new HdfsPartition instance. The old
HdfsPartition instance is only replaced at the end of the
modifications.
- Move the "dirty" marker of HdfsPartition into a map of HdfsTable. It
maps from the old partition id to the in-progress partition builder.
For these "dirty" partitions, we'll reload its HMS meta and file meta.
Tests:
- No new tests are added since the existing tests have provided
sufficient coverage
- Run CORE tests
Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c
---
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 756 insertions(+), 490 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/15985/10
--
To view, visit http://gerrit.cloudera.org:8080/15985
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib52e5810d01d5e0c910daacb9c98977426d3914c
Gerrit-Change-Number: 15985
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Anurag Mantripragada <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
