Miklos Szurap created IMPALA-6619: ------------------------------------- Summary: Alter table recover partitions creates unneeded partitions when faces percent sign Key: IMPALA-6619 URL: https://issues.apache.org/jira/browse/IMPALA-6619 Project: IMPALA Issue Type: Bug Components: Frontend Affects Versions: Impala 2.11.0 Reporter: Miklos Szurap
When a table has a partition with a special character in it's name, then the HDFS directory contains a percent sign (due to an escaped/UrlEncoded sequence). This is not decoded and compared properly when running {{alter table recover partitions}}. This creates new, unneeded partitions on each execution. The steps to reproduce/demonstrate the issue: {noformat} [nightly-2:21000] > CREATE TABLE tbl_with_partition(col1 string) partitioned by (p string); Query: CREATE TABLE tbl_with_partition(col1 string) partitioned by (p string) Fetched 0 row(s) in 1.08s [nightly-2:21000] > ALTER TABLE tbl_with_partition add partition (p='100%'); Query: ALTER TABLE tbl_with_partition add partition (p='100%') Fetched 0 row(s) in 5.72s [nightly-2:21000] > show partitions tbl_with_partition; Query: show partitions tbl_with_partition +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+ | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+ | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 | | Total | -1 | 0 | 0B | 0B | | | | | +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+ Fetched 2 row(s) in 0.02s [nightly-2:21000] > ALTER TABLE tbl_with_partition recover partitions; Query: ALTER TABLE tbl_with_partition recover partitions Fetched 0 row(s) in 0.29s [nightly-2:21000] > show partitions tbl_with_partition; Query: show partitions tbl_with_partition +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+ | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+ | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 | | 100%25 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%2525 | | Total | -1 | 0 | 0B | 0B | | | | | +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+ Fetched 3 row(s) in 0.02s [nightly-2:21000] > ALTER TABLE tbl_with_partition recover partitions; Query: ALTER TABLE tbl_with_partition recover partitions Fetched 0 row(s) in 0.27s [nightly-2:21000] > show partitions tbl_with_partition; Query: show partitions tbl_with_partition +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+ | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+ | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 | | 100%25 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%2525 | | 100%2525 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%252525 | | Total | -1 | 0 | 0B | 0B | | | | | +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+ Fetched 4 row(s) in 0.02s{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)