Quanlong Huang created IMPALA-13340:
---------------------------------------
Summary: COPY TESTCASE in LocalCatalog mode doesn't dump the
partition and file metadata
Key: IMPALA-13340
URL: https://issues.apache.org/jira/browse/IMPALA-13340
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Quanlong Huang
IMPALA-11901 fixes the failures of using COPY TESTCASE statements in
LocalCatalog mode. However, only the table metadata is dumped, e.g. table
schema, column stats. The partition and file metadata are missing.
To reproduce the issue locally, start the Impala cluster in LocalCatalog mode.
{code:bash}
bin/start-impala-cluster.py --catalogd_args=--catalog_topic_mode=minimal
--impalad_args=--use_local_catalog{code}
Dump the metadata of a query on a partitioned table:
{noformat}
copy testcase to '/tmp' select * from functional_parquet.alltypes;
+--------------------------------------------------------------------------------------+
| Test case data output path
|
+--------------------------------------------------------------------------------------+
|
hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1
|
+--------------------------------------------------------------------------------------+
{noformat}
Check the metadata from the source cluster
{noformat}
show partitions functional_parquet.alltypes
+-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
| year | month | #Rows | #Files | Size | Bytes Cached | Cache Replication
| Format | Incremental stats | Location
| EC Policy |
+-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
| 2009 | 1 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=1 |
NONE |
| 2009 | 2 | -1 | 1 | 8.09KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=2 |
NONE |
| 2009 | 3 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=3 |
NONE |
| 2009 | 4 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=4 |
NONE |
| 2009 | 5 | -1 | 1 | 8.55KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=5 |
NONE |
| 2009 | 6 | -1 | 1 | 8.23KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=6 |
NONE |
| 2009 | 7 | -1 | 1 | 8.25KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=7 |
NONE |
| 2009 | 8 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=8 |
NONE |
| 2009 | 9 | -1 | 1 | 8.41KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=9 |
NONE |
| 2009 | 10 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=10 |
NONE |
| 2009 | 11 | -1 | 1 | 8.44KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=11 |
NONE |
| 2009 | 12 | -1 | 1 | 8.59KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=12 |
NONE |
| 2010 | 1 | -1 | 1 | 8.38KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=1 |
NONE |
| 2010 | 2 | -1 | 1 | 7.80KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=2 |
NONE |
| 2010 | 3 | -1 | 1 | 8.29KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=3 |
NONE |
| 2010 | 4 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=4 |
NONE |
| 2010 | 5 | -1 | 1 | 8.62KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=5 |
NONE |
| 2010 | 6 | -1 | 1 | 8.26KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=6 |
NONE |
| 2010 | 7 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=7 |
NONE |
| 2010 | 8 | -1 | 1 | 8.64KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=8 |
NONE |
| 2010 | 9 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=9 |
NONE |
| 2010 | 10 | -1 | 1 | 8.59KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=10 |
NONE |
| 2010 | 11 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=11 |
NONE |
| 2010 | 12 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED
| PARQUET | false |
hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=12 |
NONE |
| Total | | -1 | 24 | 201.52KB | 0B |
| | |
| |
+-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+{noformat}
Import the metadata
{noformat}
copy testcase from
'hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1';
+----------------------------------------------------------------------------------------------------------------+
| summary
|
+----------------------------------------------------------------------------------------------------------------+
| Testcase generated using Impala version 4.5.0-SNAPSHOT. 1 db(s), 1 table(s)
and 0 view(s) imported for query: |
|
|
| SELECT * FROM functional_parquet.alltypes
|
+----------------------------------------------------------------------------------------------------------------+{noformat}
Check the partition metadata, they are missing:
{noformat}
set PLANNER_TESTCASE_MODE=true;
show partitions functional_parquet.alltypes;
Query: show partitions functional_parquet.alltypes
+-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
| year | month | #Rows | #Files | Size | Bytes Cached | Cache Replication |
Format | Incremental stats | Location | EC Policy |
+-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
| Total | | -1 | 0 | 0B | 0B | |
| | | |
+-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
