Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10792 )
Change subject: IMPALA-3040: Remove cache directive before dropping a table
......................................................................
Patch Set 2:
Thanks for the explanation. I guess I understand the issue now.
The basic problem here seems to be that HdfsTable.dropPartition() does not
clean up cache directives of the partitions. We do it inside
CatalogOpExecutor#alterTableDropPartition()
private Table alterTableDropPartition(Table tbl,.....)
......
if (part.isMarkedCached()) {
HdfsCachingUtil.removePartitionCacheDirective(part);
}
....
}
but since these partitions are dropped already using Hive, Impala clears the
state with dropPartition() and uncacheTable() later does not help anymore.
Here is a simple repro of this issue.
// Create a partitioned cached table and add some partitions
impala> create table cached_tbl_part (i int) partitioned by (j int) cached in
'testPool'
impala> insert into cached_tbl_part (i,j) select 1, 2;
// Make sure cache directives are created.
$ hdfs cacheadmin -listDirectives | grep cached_tbl_part
277 testPool 1 never /test-warehouse/cached_tbl_part
278 testPool 1 never /test-warehouse/cached_tbl_part/j=2 <----
// Drop the partition from hive
hive> alter table cached_tbl_part drop partition (j=2);
// Refresh the table from Impala
impala> refresh cached_tbl_part;
// Cache directives still exist
hdfs cacheadmin -listDirectives | grep cached_tbl_part
277 testPool 1 never /test-warehouse/cached_tbl_part
278 testPool 1 never /test-warehouse/cached_tbl_part/j=2 <--- should
have been dropped
// Drop the table from Impala
impala> drop table cached_tbl_part;
$ hdfs cacheadmin -listDirectives | grep cached_tbl_part
278 testPool 1 never /test-warehouse/cached_tbl_part/j=2 <---
Table's directive is dropped but the partition still remains.
--
To view, visit http://gerrit.cloudera.org:8080/10792
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170
Gerrit-Change-Number: 10792
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Tianyi Wang <[email protected]>
Gerrit-Comment-Date: Tue, 03 Jul 2018 07:43:37 +0000
Gerrit-HasComments: No
