Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14547 )
Change subject: IMPALA-9045: Filter base directories of open/aborted compactions ...................................................................... IMPALA-9045: Filter base directories of open/aborted compactions Base directories are in the format of base_<write_id>_<transaction_id>. The <transaction_id> part helps to decide whether a base directory is fully written, or is it still being written by a compaction job. Compaction jobs don't increase the write id of a table, hence the <write_id> part cannot be used for that. Before this commit Impala didn't check the validity of <transaction_id>, therefore it might read the contents of a half-written base directory. With this change Impala retrieves the valid transaction list from HMS and checks if <transaction_id> is committed. Testing * Added an e2e test that simulates in-progress compactions * Added frontend test that filters based on custom valid txn list Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Reviewed-on: http://gerrit.cloudera.org:8080/14547 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M tests/query_test/test_acid.py 12 files changed, 247 insertions(+), 44 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
