[
https://issues.apache.org/jira/browse/HADOOP-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712674#comment-16712674
]
Steve Loughran commented on HADOOP-14000:
-----------------------------------------
It's all private code, AFAIK nothing external has used it, so we aren't going
to be stepping on toes.
Now, one risk of the iterator design is that as things change underneath, you
may get an inconsistent listing back. But: that may happen anyway today, and
the listFiles() calls in HDFS is itself vulnerable to it, as is the S3A
connector. We call out this fact in the FS specs anyway "changes may happen"
> s3guard metadata stores to support millons of children
> ------------------------------------------------------
>
> Key: HADOOP-14000
> URL: https://issues.apache.org/jira/browse/HADOOP-14000
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Priority: Major
>
> S3 repos can have millions of child entries
> Currently {{DirListingMetaData}} can't and {{MetadataStore.listChildren(Path
> path)}} won't be able to handle directories that big, for listing, deleting
> or naming.
> We will need a paged response from the listing operation, something which can
> be iterated over.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
