[
https://issues.apache.org/jira/browse/ARROW-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou updated ARROW-10788:
-----------------------------------
Summary: [C++] Make S3 recursive walks parallel (was: [C++] Make S3
recursive calls parallel)
> [C++] Make S3 recursive walks parallel
> --------------------------------------
>
> Key: ARROW-10788
> URL: https://issues.apache.org/jira/browse/ARROW-10788
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Antoine Pitrou
> Assignee: Antoine Pitrou
> Priority: Major
>
> Doing a recursive S3 directory walk using GetFileInfo(Selector) currently
> lists all encountered directories serially, waiting for the results of one
> directory listing (or portion thereof) before launching the next one.
> Instead, we should use the Async APIs provided by the AWS SDK to parallelize
> HTTP requests as much as possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
