Hello Jason Fehr, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21864
to look at the new patch set (#3).
Change subject: IMPALA-13340: Fix missing partitions in COPY TESTCASE of
LocalCatalog mode
......................................................................
IMPALA-13340: Fix missing partitions in COPY TESTCASE of LocalCatalog mode
There are 3 places that we should fix:
*Exporting testcase files*
In LocalCatalog mode coordinators, to export the testcase file,
LocalFsTable objects are converted to THdfsTable objects. In this step,
coordinators should set the field of 'has_full_partitions' to true.
Otherwise, the partition map will be ignored when catalogd imports the
THdfsTable object.
*Importing testcase files*
When importing the THdfsTable object, catalogd should regenerate the
partition ids since those in the testcase file are usually generated by
the other catalogd instance (of another cluster). Reusing them might
conflict with the existing partition ids. Note that partition ids are
incremental ids generated by catalogd itself (starts from 0 at
bootstrap).
Table.loadFromThrift() is primarily used in coordinator side to load
metadata from catalogd. We always set 'storedInImpaladCatalogCache_' to
true in this method. However, this method is also used in catalogd to
import metadata from a testcase file. This patch adds a parameter to
this method to distinguish where it's used. So we can decide whether to
reuse the partition ids or generate new ones.
*Fetching metadata from catalogd*
When catalogd processes the getPartitalCatalog requests on the imported
partitions, HdfsPartition#setPartitionMetadata() is used to update the
TPartialPartitionInfo instance. Previously this method uses
'cachedMsPartitionDescriptor_ == null' to detect prototype partitions or
the only partition of unpartitioned tables. This is incorrect now since
HdfsPartitions imported from testcase files won't have
'cachedMsPartitionDescriptor_' set. The values of this field come from
msPartition objects from HMS and are not passed to the coordinators,
thus not exist in the exported testcase files. This patch fixes the
condition to check prototype partition and unpartitioned tables
correctly.
Tests
- Added e2e tests to dump the partitioned table and verify the
partition and file metadata after importing it back. The test also
verify that we can get the same query plan after importing the
testcase file.
- Moved the util method __get_partition_id_set() from
test_reuse_partitions.py to ImpalaTestSuite so we can reuse it in the
new test. Also renamed it to get_partition_id_set().
Change-Id: Icc2e8b71564ad37973ddfca92801afea8e26ff73
---
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/View.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M tests/common/impala_test_suite.py
M tests/metadata/test_reuse_partitions.py
M tests/metadata/test_testcase_builder.py
18 files changed, 151 insertions(+), 79 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21864/3
--
To view, visit http://gerrit.cloudera.org:8080/21864
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icc2e8b71564ad37973ddfca92801afea8e26ff73
Gerrit-Change-Number: 21864
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
