Aaron Fabbri commented on HADOOP-12876:

FYI we're working on on something similar for S3.  The umbrella JIRA is 
HADOOP-13448 and the in-memory implementation is HADOOP-13452.  Once this work 
is complete, it should be easy to refactor so you can access the 
LocalMetadataStore from ADLS client.

> [Azure Data Lake] Support for process level FileStatus cache to optimize 
> GetFileStatus frequent opeations
> ---------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-12876
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12876
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/azure, tools
>            Reporter: Vishwajeet Dusane
>            Assignee: Vishwajeet Dusane
> Add support to cache GetFileStatus and ListStatus response locally for 
> limited period of time. Local cache for limited period of time would optimize 
> number of calls for GetFileStatus operation.
> One of the example  where local limited period cache would be useful - 
> terasort ListStatus on input directory follows with GetFileStatus operation 
> on each file within directory. For 2048 input files in a directory would save 
> 2048 GetFileStatus calls during start up (Using the ListStatus response to 
> cache FileStatus instances).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to