pitrou opened a new pull request #8818: URL: https://github.com/apache/arrow/pull/8818
Use the AWS SDK async APIs to launch child directory reads concurrently as soon as we get the required information from a parent read. Also, similarly issue directory tree deletion commands in parallel. On this machine, listing the entire directory tree at "s3://mf-nwp-models/arome-france/v2/2020-12-02" goes down from 12 seconds to 2 seconds (a 6x speedup). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
