[
https://issues.apache.org/jira/browse/HIVE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17395105#comment-17395105
]
Sungwoo commented on HIVE-25277:
--------------------------------
In HIVE-24849, isEmpty() is rewritten to call FileSystem.listStatusIterator()
instead of FileSystem.listStatus(). If duplicate calls to isEmpty() are part of
the problem, HIVE-24849 can also help.
> Slow Hive partition deletion for Cloud object stores with expensive ListFiles
> -----------------------------------------------------------------------------
>
> Key: HIVE-25277
> URL: https://issues.apache.org/jira/browse/HIVE-25277
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Affects Versions: All Versions
> Reporter: Zhou Fang
> Assignee: Zhou Fang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Deleting a Hive partition is slow when use a Cloud object store as the
> warehouse for which ListFiles is expensive. A root cause is that the
> recursive parent dir deletion is very inefficient: there are many duplicated
> calls to isEmpty (ListFiles is called at the end). This fix sorts the parents
> to delete according to the path size, and always processes the longest one
> (e.g., a/b/c is always before a/b). As a result, each parent path is only
> needed to be checked once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
