[
https://issues.apache.org/jira/browse/IMPALA-7084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493850#comment-16493850
]
Fredy Wijaya commented on IMPALA-7084:
--------------------------------------
It doesn't seem to be an issue anymore in the latest master (3.x) and 2.x
branches.
{noformat}
[localhost:21000] default> create table test (a int) partitioned by (b int);
Query: create table test (a int) partitioned by (b int)
Fetched 1 row(s) in 0.02s
>[localhost:21000] default> insert into test partition (b=1) values (1);
Query: insert into test partition (b=1) values (1)
Query submitted at: 2018-05-29 08:35:14 (Coordinator: http://impala-dev:25000)
Query progress can be monitored at:
http://impala-dev:25000/query_plan?query_id=84adf7dc8062947:7c9176fc00000000
Modified 1 row(s) in 4.10s
[localhost:21000] default> alter table test add partition (b=2) location
'hdfs://localhost:20500/test-warehouse/test/b=2/';
Query: alter table test add partition (b=2) location
'hdfs://localhost:20500/test-warehouse/test/b=2/'
+--------------------------------------------+
| summary |
+--------------------------------------------+
| New partition has been added to the table. |
+--------------------------------------------+
Fetched 1 row(s) in 0.03s
[localhost:21000] default> alter table test partition (b=1) set location
'hdfs://localhost:20500/test-warehouse/test/b=5/';
Query: alter table test partition (b=1) set location
'hdfs://localhost:20500/test-warehouse/test/b=5/'
+--------------------------------------------------------+
| summary |
+--------------------------------------------------------+
| New location has been set for the specified partition. |
+--------------------------------------------------------+
Fetched 1 row(s) in 0.07s
[localhost:21000] default> show partitions test;
Query: show partitions test
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
| b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
Incremental stats | Location |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
| 1 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false |
hdfs://localhost:20500/test-warehouse/test/b=5
|
| 2 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false |
hdfs://localhost:20500/test-warehouse/test/b=2
|
| Total | -1 | 0 | 0B | 0B | | | |
|
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
Fetched 3 row(s) in 0.01s{noformat}
> Partition doesn't exist after attempting to ALTER partition location to
> non-existing path
> -----------------------------------------------------------------------------------------
>
> Key: IMPALA-7084
> URL: https://issues.apache.org/jira/browse/IMPALA-7084
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.12.0
> Reporter: Gabor Kaszab
> Priority: Major
> Labels: correctness, ramp-up
>
>
>
> {code:java}
> create table test (a int) partitioned by (b int);
> insert into test partition (b=1) values (1);
> // create another partition that points to a different location.
> alter table test add partition (b=2) location
> 'hdfs://localhost:20500/test-warehouse/test/b=2/';
> // setting the first partition to a non existing location. This fails as
> expected. The error message is not exactly user friendly, though. Could have
> said "Invalid location or such".
> alter table test partition (b=1) set location
> 'hdfs://localhost:20500/test-warehouse/test/b=5/';
> Query: alter table test partition (b=1) set location
> 'hdfs://localhost:20500/test-warehouse/test/b=5/'
> ERROR: TableLoadingException: Failed to load metadata for table: default.test
> CAUSED BY: NullPointerException: null
> // Setting the first partition to an existing location. This surprisingly
> fails as the partitions doesn't exist.
> alter table test partition (b=1) set location
> 'hdfs://localhost:20500/test-warehouse/test/b=2/';
> Query: alter table test partition (b=1) set location
> 'hdfs://localhost:20500/test-warehouse/test/b=1/'
> ERROR: PartitionNotFoundException: Partition not found:
> TPartitionKeyValue(name:b, value:1)
> // However, show partition displays b=1 partition as well.
> show partitions test;
> Query: show partitions test
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
> | b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
> Incremental stats | Location |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
> | 1 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT | false |
> hdfs://localhost:20500/test-warehouse/test/b=1 |
> | 2 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT | false |
> hdfs://localhost:20500/test-warehouse/test/b=2 |
> | Total | -1 | 2 | 4B | 0B | | | | |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
> // Invalidate metadata fixes this issue
> invalidate metadata test;
> // Now show partitions say that b=1 partition points to the non-existing
> location. (This is not a problem as the location would be
> // created one data is put into that partition again.)
> show partitions test;
> Query: show partitions test
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------
> --------------------------+
> | b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
> Incremental stats | Location
> |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------
> --------------------------+
> | 1 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false |
> hdfs://localhost:2050
> 0/test-warehouse/test/b=5 |
> | 2 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false |
> hdfs://localhost:2050
> 0/test-warehouse/test/b=2 |
> | Total | -1 | 0 | 0B | 0B | | | |
> |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------
> --------------------------+
> // After the invalidate metadata the partition is again visilbe for ALTER
> TABLE partition location commands.
> alter table test partition (b=1) set location
> 'hdfs://localhost:20500/test-warehouse/test/b=2/';
> Query: alter table test partition (b=1) set location
> 'hdfs://localhost:20500/test-warehouse/test/b=2/'
> +--------------------------------------------------------+
> | summary |
> +--------------------------------------------------------+
> | New location has been set for the specified partition. |
> +--------------------------------------------------------+
> // {code}
>
> My expectation here would be that in case the first alter partition location
> command fails due to non-existing location then no state change should be
> done in the background. In addition seeing an existing partition as a
> non-existing one after a failed attempt to change some metadata is also
> undesired.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
