[
https://issues.apache.org/jira/browse/HADOOP-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157003#comment-14157003
]
Colin Patrick McCabe commented on HADOOP-10987:
-----------------------------------------------
{code}
// Each file system implementation should override this method and
// provide more efficient implementation, if possible.
{code}
Hmm, this might look better in the JavaDoc header.
{code}
+ public RemoteIterator<FileStatus> listStatusIterator(final Path p)
+ throws FileNotFoundException, IOException {
+ return listStatusIterator(p, DEFAULT_FILTER);
+ }
{code}
Should this method be final? It seems like we'll always want it to act like
the {{listStatusIterator}} method with {{DEFAULT_FILTER}}.
{code}
- protected RemoteIterator<LocatedFileStatus> listLocatedStatus(final Path f,
+ public RemoteIterator<LocatedFileStatus> listLocatedStatus(final Path f,
final PathFilter filter)
{code}
I like the fact that you changed more of our subclasses to implement the
two-argument form of {{listLocatedStatus}}. They should definitely do this
when possible (like FilterFileSystem should pass the two-argument form to what
it contains).
But it seems like changing the visibility from {{protected}} to {{public}} will
be a slight incompatibility for some out-of-tree filesystems. If they
previously overrode the protected two-argument form with a protected function,
they will get a compile error now: "cannot reduce visibility of inherited
method." Maybe we can just leave the two-argument form of
{{listLocatedStatus}} as {{protected}} for now, and handle it in a follow-on
JIRA? Just adding {{listStatusIterator}} is already a pretty nice change.
> Provide an iterator-based listing API for FileSystem
> ----------------------------------------------------
>
> Key: HADOOP-10987
> URL: https://issues.apache.org/jira/browse/HADOOP-10987
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Attachments: HADOOP-10987.patch, HADOOP-10987.v2.patch,
> HADOOP-10987.v3.patch
>
>
> Iterator based listing methods already exist in {{FileContext}} for both
> simple listing and listing with locations. However, {{FileSystem}} lacks the
> former. From what I understand, it wasn't added to {{FileSystem}} because it
> was believed to be phased out soon. Since {{FileSystem}} is very well alive
> today and new features are getting added frequently, I propose adding an
> iterator based {{listStatus}} method. As for the name of the new method, we
> can use the same name used in {{FileContext}} : {{listStatusIterator()}}.
> It will be particularly useful when listing giant directories. Without this,
> the client has to build up a huge data structure and hold it in memory. We've
> seen client JVMs running out of memory because of this.
> Once this change is made, we can modify FsShell, etc. in followup jiras.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
