Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14547
to look at the new patch set (#3).
Change subject: IMPALA-9045: Filter base directories of open/aborted compactions
......................................................................
IMPALA-9045: Filter base directories of open/aborted compactions
Base directories are in the format of base_<write_id>_<transaction_id>.
The <transaction_id> part helps to decide whether a base directory
is fully written, or is it still being written by a compaction job.
Compaction jobs don't increase the write id of a table, hence the
<write_id> part cannot be used for that.
Before this commit Impala didn't check the validity of <transaction_id>,
therefore it might read the contents of a half-written base directory.
With this change Impala retrieves the valid transaction list from HMS
and checks if <transaction_id> is committed.
Testing
* Added an e2e test that simulates in-progress compactions
* Added frontend test that filters based on custom valid txn list
Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6
---
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java
M tests/query_test/test_acid.py
12 files changed, 246 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/14547/3
--
To view, visit http://gerrit.cloudera.org:8080/14547
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6
Gerrit-Change-Number: 14547
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
