kszlim commented on issue #5378: URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-1936407201
> means you are hitting rate limits and should reduce the amount of concurrent IO you are performing. There is a LimitStore Hmm, I've tried reducing concurrency in polars (to 32) and it doesn't seem to fix this, though I doubt i'm anywhere near the stated rate limit of s3 which is 5500 GET requests per partitioned prefix per second. If it's not incorrect behavior, do you think it makes sense to also build this condition into the retries done by object store @tustvold ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
