[
https://issues.apache.org/jira/browse/HIVE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17395114#comment-17395114
]
Zhou Fang commented on HIVE-25277:
----------------------------------
[~glapark] According to listStatusIterator's implementation, it calls
`listStatus` to get all statuses and caches the result. listStatus is still
expensive for Cloud object stores, so I think it does not solve the problem.
private FileStatus[] statusList = listStatus(f);
[https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java#L944|https://github.com/apache/hadoop/blob/release-3.3.1-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java#L945]
> Slow Hive partition deletion for Cloud object stores with expensive ListFiles
> -----------------------------------------------------------------------------
>
> Key: HIVE-25277
> URL: https://issues.apache.org/jira/browse/HIVE-25277
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Affects Versions: All Versions
> Reporter: Zhou Fang
> Assignee: Zhou Fang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Deleting a Hive partition is slow when use a Cloud object store as the
> warehouse for which ListFiles is expensive. A root cause is that the
> recursive parent dir deletion is very inefficient: there are many duplicated
> calls to isEmpty (ListFiles is called at the end). This fix sorts the parents
> to delete according to the path size, and always processes the longest one
> (e.g., a/b/c is always before a/b). As a result, each parent path is only
> needed to be checked once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
