[
https://issues.apache.org/jira/browse/HDFS-10784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435496#comment-15435496
]
Xiao Chen commented on HDFS-10784:
----------------------------------
Thanks Andrew for working on this, and Brahma for bringing up HDFS-9366. Patch
itself looks good.
The feature the 2 jiras want to solve looks very alike - allowing some
pagination when listing. Implementation is a little different:
- HDFS-10784 implements {{RemoteIterator}}, so adds a new interface
{{listStatusIterator}}.
- HDFS-9366 overloads current {{listStatus}}, with customized offset and size
parameters.
IMHO, HDFS-10784 is cleaner and more flexible, hence easier to use when user
wants to iterate the whole listing. HDFS-9366 could have less end-to-end trips
when listing with a starting offset not at the beginning.
Looking at {{DistributedFileSystem}}, an iterator would be more consistent with
hdfs context. I don't have a strong opinion, maybe we should ask our user
[~romainr] since both jira seems to aim at Hue.
I think we should combine this and HDFS-9366 after agreement, and add
documentation. Would be great if httpfs is supported too.
> Implement WebHdfsFileSystem#listStatusIterator
> ----------------------------------------------
>
> Key: HDFS-10784
> URL: https://issues.apache.org/jira/browse/HDFS-10784
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
> Affects Versions: 2.6.4
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: HDFS-10784.001.patch
>
>
> It would be nice to implement the iterative listStatus in WebHDFS so client
> apps do not need to buffer the full file list for large directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
