Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21864
to look at the new patch set (#2).
Change subject: IMPALA-13340: Fix missing partitions in COPY TESTCASE of
LocalCatalog mode
......................................................................
IMPALA-13340: Fix missing partitions in COPY TESTCASE of LocalCatalog mode
There are 3 places that we should fix:
*Exporting testcase files*
In LocalCatalog mode coordinators, to export the testcase file,
LocalFsTable objects are converted to THdfsTable objects. In this step,
coordinators should to set the field of 'has_full_partitions' to true.
Otherwise, the partition map will be ignored when catalogd imports the
THdfsTable object.
*Importing testcase files*
When importing the THdfsTable object, catalogd should regenerate the
partition ids since those in the testcase file are usually generated by
the other catalogd instance (of another cluster). Reusing them might
conflict with the existing partition ids. Note that partition ids are
incremental ids generated by catalogd itself (starts from 0 at
bootstrap).
Table.loadFromThrift() is primarily used in coordinator side to load
metadata from catalogd. We always set 'storedInImpaladCatalogCache_' to
true in this method. However, this method is also used in catalogd to
import metadata from a testcase file. This patch adds a parameter to
this method to distinguish where it's used. So we can decide whether to
reuse the partition ids or generate new ones.
*Fetching metadata from catalogd*
When catalogd processes the getPartitalCatalog requests on the imported
partitions, HdfsPartition#setPartitionMetadata() is used to update the
TPartialPartitionInfo instance. Previously this method uses
'cachedMsPartitionDescriptor_ == null' to detect prototype partitions or
the only partition of unpartitioned tables. This is incorrect now since
HdfsPartitions imported from testcase files won't have
'cachedMsPartitionDescriptor_' set. The values of this field come from
msPartition objects from HMS and are not passed to the coordinators,
thus not exist in the exported testcase files. This patch fixes the
condition to check prototype partition and unpartitiond tables
correctly.
Tests
- Add e2e tests to dump the partitioned table and verify the partition
and file metadata after importing it back. Also verify that we can
get the same query plan after importing the testcase file.
Change-Id: Icc2e8b71564ad37973ddfca92801afea8e26ff73
---
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/View.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M tests/metadata/test_testcase_builder.py
16 files changed, 116 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21864/2
--
To view, visit http://gerrit.cloudera.org:8080/21864
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icc2e8b71564ad37973ddfca92801afea8e26ff73
Gerrit-Change-Number: 21864
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
