olchas edited a comment on pull request #11284: URL: https://github.com/apache/airflow/pull/11284#issuecomment-704180364
> Should we make this configurable to avoid breaking change? I see the point of downloading a single file but this may be problematic when downloading multiple files with the same name (for example structure like `output/2020-09-20/data.csv` and downloading data for multiple days). WDYT @TobKed ? From what I understood by reading the code, only the part up to the wildcard character is stripped from the `source_object` path. In other word, when downloading multiple files from your example, you would set `source_object` to something like this: `output/*`. Then, when uploading them to `destination_path`, only the `output` would be stripped from every collected object's path, so you should get multiple directories for multiple days in your destination. Did I get this right, @TobKed? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
