Venugopal Reddy K created IMPALA-13141:
------------------------------------------
Summary: Partition transactional table is not updated on alter
partition when hms_event_incremental_refresh_transactional_table is disabled
Key: IMPALA-13141
URL: https://issues.apache.org/jira/browse/IMPALA-13141
Project: IMPALA
Issue Type: Bug
Reporter: Venugopal Reddy K
Partition transactional table is not updated on alter partition when
hms_event_incremental_refresh_transactional_table is disabled.
*Observations:*
1. In case of AlterPartitionEvent, this issue occurs when
hms_event_incremental_refresh_transactional_table is disabled.
2. In case of BatchPartitionEvent(when. more than 1
AlterPartitionEvent are batched together), this issue occurs without disabling
hms_event_incremental_refresh_transactional_table.
*Steps to reproduce:*
1. Create partitioned table and add some partitions from hive:
Note: This step can be from impala too.
{code:java}
0: jdbc:hive2://localhost:11050> create table s(i int, j int, p int);
0: jdbc:hive2://localhost:11050> insert into s values(1,10,100),(2,20,200);
{code}
{code:java}
0: jdbc:hive2://localhost:11050> create table test1(i int, j int) partitioned
by(p int) tblproperties ('transactional'='true',
'transactional_properties'='insert_only');
0: jdbc:hive2://localhost:11050> set hive.exec.dynamic.partition.mode=nonstrict;
0: jdbc:hive2://localhost:11050> insert into test partition(p) select * from s;
0: jdbc:hive2://localhost:11050> show partitions test;
+------------+
| partition |
+------------+
| p=100 |
| p=200 |
+------------+
0: jdbc:hive2://localhost:11050> desc formatted test partition(p=100);
+-----------------------------------+----------------------------------------------------+-----------------------+
| col_name | data_type
| comment |
+-----------------------------------+----------------------------------------------------+-----------------------+
| i | int
| |
| j | int
| |
| | NULL
| NULL |
| # Partition Information | NULL
| NULL |
| # col_name | data_type
| comment |
| p | int
| |
| | NULL
| NULL |
| # Detailed Partition Information | NULL
| NULL |
| Partition Value: | [100]
| NULL |
| Database: | default
| NULL |
| Table: | test
| NULL |
| CreateTime: | Fri Jun 07 14:21:17 IST 2024
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Location: |
hdfs://localhost:20500/test-warehouse/managed/test/p=100 | NULL
|
| Partition Parameters: | NULL
| NULL |
| | numFiles
| 1 |
| | totalSize
| 5 |
| | transient_lastDdlTime
| 1717750277 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: |
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat
| NULL |
| OutputFormat: |
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | -1
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
| Storage Desc Params: | NULL
| NULL |
| | serialization.format
| 1 |
+-----------------------------------+----------------------------------------------------+-----------------------+
0: jdbc:hive2://localhost:11050>
{code}
2. Check for the partitions and its locations from impala shell:
{code:java}
[localhost:21050] default> show partitions test;
Query: show partitions test
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------+-----------+
| p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
Incremental stats | Location |
EC Policy |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------+-----------+
| 100 | -1 | 1 | 5B | NOT CACHED | NOT CACHED | TEXT |
false | hdfs://localhost:20500/test-warehouse/managed/test/p=100 |
NONE |
| 200 | -1 | 1 | 5B | NOT CACHED | NOT CACHED | TEXT |
false | hdfs://localhost:20500/test-warehouse/managed/test/p=200 |
NONE |
| Total | -1 | 2 | 10B | 0B | | |
| |
|
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------+-----------+
Fetched 3 row(s) in 4.43s
[localhost:21050] default>
[localhost:21050] default> select * from test;
Query: select * from test
Query submitted at: 2024-06-07 14:28:13 (Coordinator:
http://kvenureddy-OptiPlex-7000:25000)
Query state can be monitored at:
http://kvenureddy-OptiPlex-7000:25000/query_plan?query_id=024dff2843ae25bf:6d5792d100000000
+---+----+-----+
| i | j | p |
+---+----+-----+
| 2 | 20 | 200 |
| 1 | 10 | 100 |
+---+----+-----+
Fetched 2 row(s) in 0.34s
[localhost:21050] default> select * from test where p=100;
Query: select * from test where p=100
Query submitted at: 2024-06-07 14:28:19 (Coordinator:
http://kvenureddy-OptiPlex-7000:25000)
Query state can be monitored at:
http://kvenureddy-OptiPlex-7000:25000/query_plan?query_id=fa42921d722ecc8a:6e7e545400000000
+---+----+-----+
| i | j | p |
+---+----+-----+
| 1 | 10 | 100 |
+---+----+-----+
Fetched 1 row(s) in 0.11s
[localhost:21050] default>
{code}
3. Alter partition location from hive:
{code:java}
0: jdbc:hive2://localhost:11050> alter table test1 partition(p=100) set
location '/dummylocation100';
0: jdbc:hive2://localhost:11050> desc formatted test partition(p=100);
+-----------------------------------+----------------------------------------------------+-----------------------+
| col_name | data_type
| comment |
+-----------------------------------+----------------------------------------------------+-----------------------+
| i | int
| |
| j | int
| |
| | NULL
| NULL |
| # Partition Information | NULL
| NULL |
| # col_name | data_type
| comment |
| p | int
| |
| | NULL
| NULL |
| # Detailed Partition Information | NULL
| NULL |
| Partition Value: | [100]
| NULL |
| Database: | default
| NULL |
| Table: | test
| NULL |
| CreateTime: | Fri Jun 07 14:21:17 IST 2024
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Location: | hdfs://localhost:20500/dummylocation100
| NULL |
| Partition Parameters: | NULL
| NULL |
| | last_modified_by
| kvenureddy |
| | last_modified_time
| 1717750893 |
| | numFiles
| 1 |
| | totalSize
| 5 |
| | transient_lastDdlTime
| 1717750893 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: |
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat
| NULL |
| OutputFormat: |
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | -1
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
| Storage Desc Params: | NULL
| NULL |
| | serialization.format
| 1 |
+-----------------------------------+----------------------------------------------------+-----------------------+
0: jdbc:hive2://localhost:11050> {code}
4. Check for the partition location from impala shell. It is not updated.
{code:java}
[localhost:21050] default> show partitions test;
Query: show partitions test
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------+-----------+
| p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
Incremental stats | Location |
EC Policy |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------+-----------+
| 100 | -1 | 1 | 5B | NOT CACHED | NOT CACHED | TEXT |
false | hdfs://localhost:20500/test-warehouse/managed/test/p=100 |
NONE |
| 200 | -1 | 1 | 5B | NOT CACHED | NOT CACHED | TEXT |
false | hdfs://localhost:20500/test-warehouse/managed/test/p=200 |
NONE |
| Total | -1 | 2 | 10B | 0B | | |
| |
|
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------+-----------+
Fetched 3 row(s) in 0.01s
[localhost:21050] default>
[localhost:21050] default> select * from test where p=100;
Query: select * from test where p=100
Query submitted at: 2024-06-07 14:35:06 (Coordinator:
http://kvenureddy-OptiPlex-7000:25000)
Query state can be monitored at:
http://kvenureddy-OptiPlex-7000:25000/query_plan?query_id=7042fda005972a2b:a82fd60700000000
+---+----+-----+
| i | j | p |
+---+----+-----+
| 1 | 10 | 100 |
+---+----+-----+
Fetched 1 row(s) in 0.11s
[localhost:21050] default>
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
