westonpace commented on code in PR #13796:
URL: https://github.com/apache/arrow/pull/13796#discussion_r939107262
##########
cpp/src/arrow/filesystem/filesystem.h:
##########
@@ -132,6 +133,9 @@ struct ARROW_EXPORT FileSelector {
bool recursive;
/// The maximum number of subdirectories to recurse into.
int32_t max_recursion;
+ /// How many partitions should be processed in parallel. May not be
supported by all
+ /// implementations of `GetFileSystemGenerator`.
+ util::optional<int32_t> partitions_readahead;
Review Comment:
Hmm...this makes sense but we haven't used `util::optional` in public
interfaces that I know of. Typically we do something like `int32_t
partitions_readahead = kDefaultPartitionsReadahead`. Although, the downside is
that we usually end up having to repeat the default in python. Curious if
@pitrou has any opinion here.
Also, I don't think "partitions" is the correct terminology to be using
here. The `FileSelector` is to be understood in the context of "filesystems"
which is a more generic abstraction than partitions. Perhaps
`directory_readahead`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
