[
https://issues.apache.org/jira/browse/IO-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215535#comment-17215535
]
Gary D. Gregory edited comment on IO-597 at 10/16/20, 5:12 PM:
---------------------------------------------------------------
Note that the current pre-Java 8 implementation of our iterateFiles APIs may
feel backward because the underlying API returns a collection of files, and our
API that returns an iterator simply iterates over this collection.
With Java NIO's Files.walk() APIs we can walk the file tree one element at a
time.
But, this is not returning an object where the call site can implement
streaming over the elements either which is what an instance of Iterator or
better, Stream does.
In order to return a Stream<Path> (or even an Iterator<File>), we need to turn
file walking inside out from a the visitor pattern to streaming where the call
to get the next item from the stream, "walks" one element "forward" or "down"
(for a directory).
This new code can all be NIO based with a bridge to IO as [~Spoor] mentions
above.
was (Author: garydgregory):
Note that the current pre-Java 8 implementation of our iterateFiles APIs may
feel backward because the underlying API returns a collection of files, and our
API that returns an iterator simply iterates over this collection.
With Java NIO's Files.walk() APIs we can walk the file tree one element at a
time.
But, this is not returning an object where the call site can implement
streaming over the elements either which is what an instance of Iterator or
better, Stream does.
In order to return a Stream<Path> (or even an Iterator<File>), we need to turn
file walking inside out from a the visitor pattern to streaming where call to
get the next item in the stream, "walks" one element "forward" or "down" (for a
directory).
This new code can all be NIO based with a bridge to IO as [~Spoor] mentions
above.
> FileUtils.iterateFiles goes out of memory when executed for a directory with
> large number of files
> --------------------------------------------------------------------------------------------------
>
> Key: IO-597
> URL: https://issues.apache.org/jira/browse/IO-597
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
> Reporter: Arvind
> Priority: Major
>
> FileUtils.iterateFiles goes out of memory when executed for a directory with
> large number of files because it uses the listFiles method which returns an
> array of java.io.File objects. The iterator itself should not be derived from
> a list but from a Java Stream which will have lesser memory footprint. This
> feature however can be used only with Java 8 or later because streams were
> introduced only in Java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
