Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18043 )
Change subject: IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction ...................................................................... IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction After compaction happened in Hive(HIVE ACID table), queries made in Impala possibly fail with a FileNotFoundException if files already removed by the Hive cleaner. In IMPALA-10801, catalogd checks the latest compaction id before serving metadata. However, coordinators don't take advantage of that. Coordinators have their own local cache, so we will have to do the same check for coordinators as well. Besides, we also need to attach writeIdList to requests that need to fetch file metadata. Since this checking brings additional overhead for queries, we introduce a flag auto_check_compaction and set it as false by default for now. We will find some other efficient ways to do compaction checking in the future. Tests: Added unit tests to CatalogdMetaProviderTest Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b Reviewed-on: http://gerrit.cloudera.org:8080/18043 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CompactionInfoLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java 12 files changed, 356 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b Gerrit-Change-Number: 18043 Gerrit-PatchSet: 7 Gerrit-Owner: Yu-Wen Lai <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sourabh Goyal <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Yu-Wen Lai <[email protected]>
