[
https://issues.apache.org/jira/browse/HADOOP-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951839#comment-15951839
]
Mingliang Liu commented on HADOOP-13926:
----------------------------------------
Thanks, [~fabbri]!
{quote}
For unblocking merge to trunk, how about the caveat that S3Guard list
consistency does not support listFiles() yet? You simply get S3 results without
additional consistency guarantees and we'd implement listFiles() after merge.
{quote}
I'm OK with that idea. I file a new JIRA [HADOOP-14266] to track effort of
employing MetadataStore in {{S3AFileSystem::listFiles()}}. Supporting
listFiles(recursive=true) many many children is indeed challenging. Make this
JIRA focus on the {{listLocatedStatus()}} FileSystem API.
I'll update the patch to union results and populate to MS for the case of
non-authoritative. I was thinking similar ideas to what you suggested: save the
list while iterating S3 objects and populate in the end. A set will be our
helper. We may optimize this part later if reading from MetadataStore is also
w/ paging. By now, I think it's OK to use this straightforward algorithm.
Joint effort will be (as always) appreciated!
> S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore
> -------------------------------------------------------------------
>
> Key: HADOOP-13926
> URL: https://issues.apache.org/jira/browse/HADOOP-13926
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Assignee: Steve Loughran
> Attachments: HADOOP-13926-HADOOP-13345.001.patch,
> HADOOP-13926.wip.proto.branch-13345.1.patch
>
>
> Need to check if {{listLocatedStatus}} can make use of metastore's
> listChildren feature.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
