amoeba opened a new issue, #34892: URL: https://github.com/apache/arrow/issues/34892
### Describe the enhancement requested In the current implementation of at least the S3 filesystem, it's possible to hit AWS' rate limiting and this makes it hard to use a function like `pyarrow.parquet.write_dataset` to write highly partitioned datasets to S3. When the rate limit is hit during writing, the user receives an HTTP 503 with a "SLOW_DOWN" error and the request needs to be retried. I would guess any other cloud storage provider (ie GCS) would behave similarly though I haven't tested. There are a few things that could be done to address this issue: - Add logic to the S3 implementation to deal with the issue directly - Implement a generic `ThrottledFilesystem` that can wrap another `Filesystem` and throttle certain calls (ie writes) - Expose the existing `SlowFileSystem` to Python/R and possibly tweak its implementation to suit this use case ### Component(s) C++ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
