ManManson commented on code in PR #13796:
URL: https://github.com/apache/arrow/pull/13796#discussion_r939889161
##########
cpp/src/arrow/filesystem/localfs.cc:
##########
@@ -309,6 +313,202 @@ Result<std::vector<FileInfo>>
LocalFileSystem::GetFileInfo(const FileSelector& s
return results;
}
+namespace {
+
+/// Workhorse for streaming async implementation of `GetFileInfo`
+/// (`GetFileInfoGenerator`).
+///
+/// There are two variants of async discovery functions suported:
+/// 1. `DiscoverPartitionFiles`, which parallelizes traversal of individual
directories
+/// so that each directory results are yielded as a separate
`FileInfoGenerator` via
+/// an underlying `DiscoveryImplIterator`, which delivers items in chunks
(default size
+/// is `kBatchSize == 1K` items).
+/// 2. `DiscoverPartitionsFlattened`, which forwards execution to the
+/// `DiscoverPartitionFiles`, with the difference that the results from
individual
+/// sub-directory iterators are merged into the single FileInfoGenerator
stream.
+///
+/// The implementation makes use of additional attributes in `FileSelector`,
+/// such as `partitions_readahead`, which can be used to tune algorithm
+/// behavior and adjust how many directories can be processed in parallel.
+/// This option is disabled by default, so that individual partitions are
processed
+/// in serial manner via `MakeConcatenatedGenerator` under the hood.
+class AsyncStatSelector {
+ public:
+ using FileInfoGeneratorProducer = PushGenerator<FileInfoGenerator>::Producer;
Review Comment:
Surely this can be done, but I'll need to play with the code a little bit to
figure out how this will work out.
I suggest we move forward with the current approach and, in case the code
can be reshaped in a more optimal way, just provide the fix as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
