ManManson commented on code in PR #13796:
URL: https://github.com/apache/arrow/pull/13796#discussion_r946818311
##########
cpp/src/arrow/filesystem/localfs.cc:
##########
@@ -309,6 +314,240 @@ Result<std::vector<FileInfo>>
LocalFileSystem::GetFileInfo(const FileSelector& s
return results;
}
+namespace {
+
+/// Workhorse for streaming async implementation of `GetFileInfo`
+/// (`GetFileInfoGenerator`).
+///
+/// There are two variants of async discovery functions suported:
+/// 1. `DiscoverDirectoryFiles`, which parallelizes traversal of individual
directories
+/// so that each directory results are yielded as a separate
`FileInfoGenerator` via
+/// an underlying `DiscoveryImplIterator`, which delivers items in chunks
(default size
+/// is 1K items).
+/// 2. `DiscoverDirectoriesFlattened`, which forwards execution to the
+/// `DiscoverDirectoryFiles`, with the difference that the results from
individual
+/// sub-directory iterators are merged into the single FileInfoGenerator
stream.
+///
+/// The implementation makes use of additional attributes in
`LocalFileSystemOptions`,
+/// such as `directory_readahead`, which can be used to tune algorithm
+/// behavior and adjust how many directories can be processed in parallel.
+/// This option is disabled by default, so that individual directories are
processed
+/// in serial manner via `MakeConcatenatedGenerator` under the hood.
+class AsyncStatSelector {
+ public:
+ using FileInfoGeneratorProducer = PushGenerator<FileInfoGenerator>::Producer;
+
+ /// Helper class, which wraps the producer instance and ensures
+ /// that it stays "open" (i.e. `producer.is_closed() == false`)
+ /// until all discovery iterators are exhausted.
+ ///
+ /// The calss maintains a shared `State`, which wraps the original producer
+ /// and automatically calls `Close()` on it once the ref-count for the
+ /// state reaches zero (which is equivalent to finishing the file discovery
+ /// process).
+ class AutoClosingProducer {
Review Comment:
Good point, thanks for catching it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
