Dawnpool commented on PR #44126: URL: https://github.com/apache/airflow/pull/44126#issuecomment-2566131567
Hi @Dev-iL! Yes, just as you said, files are downloaded sequentially since it simply uses multiple calls to the `retrieve_file` function in the order they are collected. I totally agree with your point this approach would be inefficient in the case you said. It would be awesome if we could download multiple files concurrently, but I'm not sure what kind of approach to take (probably multi-threading?) and we should definitely do performance checks to compare after implementing the method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
