Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16008 )
Change subject: IMPALA-9791: Support validWriteIdList in getPartialCatalogObject API ...................................................................... IMPALA-9791: Support validWriteIdList in getPartialCatalogObject API This change enhances the Catalog-v2 API getPartialCatalogObject to support ValidWriteIdList as an optional field in the TableInfoSelector. When such a field is provided by the clients, catalog compares the provided ValidWriteIdList with the cached ValidWriteIdList of the table. The catalog reloads the table if it determines that the cached table is stale with respect to the ValidWriteIdList provided. In case the table is already at or above the requested ValidWriteIdList catalog uses the cached table metadata information to filter out filedescriptors pertaining to the provided ValidWriteIdList. Note that in case compactions it is possible that the requested ValidWriteIdList cannot be satisfied using the cached file-metadata for some partitions. For such partitions, catalog re-fetches the file-metadata from the FileSystem. This is helpful to provide a snapshot view of the table metadata during query compilation with respect to other changes happening to the table concurrently. Note that this change does not implement the coordinator side changes needed for catalog clients to use such a field. That would be taken up in a separate change to keep this patch smaller. Testing: 1. Ran existing filemetadata loader tests. 2. Added a new test which exercises the various cases for ValidWriteIdList comparison. Change-Id: Ied2c7c3cb2009c407e8fbc3af4722b0d34f57c4a --- M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java A fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M shaded-deps/pom.xml 16 files changed, 1,074 insertions(+), 142 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16008/4 -- To view, visit http://gerrit.cloudera.org:8080/16008 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ied2c7c3cb2009c407e8fbc3af4722b0d34f57c4a Gerrit-Change-Number: 16008 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
