Quanlong Huang created IMPALA-12356:
---------------------------------------
Summary: Partition created by INSERT will make the next
ALTER_PARTITION event on it always treated as self-event
Key: IMPALA-12356
URL: https://issues.apache.org/jira/browse/IMPALA-12356
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Quanlong Huang
In Impala, create a partitioned table and create one partition in it using
{*}INSERT{*}:
{code:sql}
create table my_part (i int) partitioned by (p int) stored as parquet;
insert into my_part partition(p=0) values (0),(1),(2);
show partitions my_part
+-------+-------+--------+------+--------------+-------------------+---------+-------------------+---------------------------------------------------+-----------+
| p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
Incremental stats | Location | EC
Policy |
+-------+-------+--------+------+--------------+-------------------+---------+-------------------+---------------------------------------------------+-----------+
| 0 | -1 | 1 | 358B | NOT CACHED | NOT CACHED | PARQUET |
false | hdfs://localhost:20500/test-warehouse/my_part/p=0 | NONE
|
| Total | -1 | 1 | 358B | 0B | | |
| |
|
+-------+-------+--------+------+--------------+-------------------+---------+-------------------+---------------------------------------------------+-----------+
{code}
In Hive, describe the partition. We can see parameters of
"impala.events.catalogServiceId" and "impala.events.catalogVersion" added by
Impala. This is ok.
{noformat}
hive> desc formatted my_part partition(p=0);
+-----------------------------------+----------------------------------------------------+-----------------------------------+
| col_name | data_type
| comment |
+-----------------------------------+----------------------------------------------------+-----------------------------------+
| i | int
| |
| | NULL
| NULL |
| # Partition Information | NULL
| NULL |
| # col_name | data_type
| comment |
| p | int
| |
| | NULL
| NULL |
| # Detailed Partition Information | NULL
| NULL |
| Partition Value: | [0]
| NULL |
| Database: | default
| NULL |
| Table: | my_part
| NULL |
| CreateTime: | Wed Aug 09 15:24:50 CST 2023
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Location: |
hdfs://localhost:20500/test-warehouse/my_part/p=0 | NULL
|
| Partition Parameters: | NULL
| NULL |
| | impala.events.catalogServiceId
| eab33ebb8a14cfd:8b2bdc12df3568df |
| | impala.events.catalogVersion
| 1882 |
| | numFiles
| 1 |
| | totalSize
| 358 |
| | transient_lastDdlTime
| 1691565890 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: |
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | NULL
|
| InputFormat: |
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat | NULL
|
| OutputFormat: |
org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | 0
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
+-----------------------------------+----------------------------------------------------+-----------------------------------+
{noformat}
Now run an ALTER statement on the partition in Hive, e.g. changing the location:
{code:sql}
alter table my_part partition(p=0) set location '/tmp';{code}
Impala will skip the ALTER_PARTITION event since it's considered as a
self-event. In catalogd logs:
{noformat}
I0809 15:30:19.628449 29844 MetastoreEvents.java:628] EventId: 8351549
EventType: ALTER_PARTITION Incremented events skipped counter to 12
I0809 15:30:19.628616 29844 MetastoreEvents.java:628] EventId: 8351549
EventType: ALTER_PARTITION Not processing the event as it is a
self-event{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
