Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21864 )
Change subject: IMPALA-13340: Fix missing partitions in COPY TESTCASE of LocalCatalog mode ...................................................................... IMPALA-13340: Fix missing partitions in COPY TESTCASE of LocalCatalog mode There are 3 places that we should fix: *Exporting testcase files* In LocalCatalog mode coordinators, to export the testcase file, LocalFsTable objects are converted to THdfsTable objects. In this step, coordinators should set the field of 'has_full_partitions' to true. Otherwise, the partition map will be ignored when catalogd imports the THdfsTable object. *Importing testcase files* When importing the THdfsTable object, catalogd should regenerate the partition ids since those in the testcase file are usually generated by the other catalogd instance (of another cluster). Reusing them might conflict with the existing partition ids. Note that partition ids are incremental ids generated by catalogd itself (starts from 0 at bootstrap). Table.loadFromThrift() is primarily used in coordinator side to load metadata from catalogd. We always set 'storedInImpaladCatalogCache_' to true in this method. However, this method is also used in catalogd to import metadata from a testcase file. This patch adds a parameter to this method to distinguish where it's used. So we can decide whether to reuse the partition ids or generate new ones. *Fetching metadata from catalogd* When catalogd processes the getPartialCatalog requests on the imported partitions, HdfsPartition#setPartitionMetadata() is used to update the TPartialPartitionInfo instance. Previously this method used 'cachedMsPartitionDescriptor_ == null' to detect prototype partitions or the only partition of unpartitioned tables. This is incorrect now since HdfsPartitions imported from testcase files won't have 'cachedMsPartitionDescriptor_' set. The values of this field come from msPartition objects from HMS and are not passed to the coordinators, thus do not exist in the exported testcase files. This patch fixes the condition to check prototype partition and unpartitioned tables correctly. Tests - Added e2e tests to dump the partitioned table and verify the partition and file metadata after importing it back. The test also verify that we can get the same query plan after importing the testcase file. - Moved the util method __get_partition_id_set() from test_reuse_partitions.py to ImpalaTestSuite so we can reuse it in the new test. Also renamed it to get_partition_id_set(). Change-Id: Icc2e8b71564ad37973ddfca92801afea8e26ff73 Reviewed-on: http://gerrit.cloudera.org:8080/21864 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M tests/common/impala_test_suite.py M tests/metadata/test_reuse_partitions.py M tests/metadata/test_testcase_builder.py 11 files changed, 126 insertions(+), 62 deletions(-) Approvals: Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icc2e8b71564ad37973ddfca92801afea8e26ff73 Gerrit-Change-Number: 21864 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
