[ 
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778983#comment-16778983
 ] 

Gopal V commented on HIVE-21225:
--------------------------------

The specific improvement I'm looking at is to make the 
{{List<HdfsFileStatusWithId> childrenWithId}} in getAcidState to be a recursive 
listing holder, which is a local change to a single function.

Eugene's suggestion is a bit better for skipping one of the looks ups, but the 
crucial one I'm looking at is to have that dir listing be all-depth instead of 
single level (FYI, all-depth listings are faster in S3 than one level deep, 
since the recursive one is just a prefix-lookup, while the one-level is a 
prefix-lookup + discard anything which has another "/" later).

> ACID: getAcidState() should cache a recursive dir listing locally
> -----------------------------------------------------------------
>
>                 Key: HIVE-21225
>                 URL: https://issues.apache.org/jira/browse/HIVE-21225
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>            Reporter: Gopal V
>            Assignee: Vaibhav Gumashta
>            Priority: Major
>         Attachments: async-pid-44-2.svg
>
>
> Currently getAcidState() makes 3 calls into the FS api which could be 
> answered by making a single recursive listDir call and reusing the same data 
> to check for isRawFormat() and isValidBase().
> All delta operations for a single partition can go against a single listed 
> directory snapshot instead of interacting with the NameNode or ObjectStore 
> within the inner loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to