westonpace commented on issue #5545: URL: https://github.com/apache/arrow-rs/issues/5545#issuecomment-2037707056
> Could you provide a bit of context on how you're running into this issue. I encountered this performing a `GET` request (part of `ObjectStore::head`) to fetch an objects size. We get failures like this very infrequently so I'm assuming it is just an exceptional event on the cloud storage servers. We used to have this problem more reliably with `get_range` but we ended up surrounding `get_range` with our own retry loop (we now also surround `size` with a similar retry loop so this isn't urgent for us). > I wonder if you are multiplexing CPU bound tasks on the same threadpool and thereby starving out the IO tasks? Yes, we do perform quite a few tasks in parallel, but we do quite a bit of profiling and CPU bound tasks shouldn't generally be starving the thread pool. > Perhaps you might file an upstream issue in the hyper repo to get feedback on exposing this upstream? I don't have any evidence to believe that `hyper` is doing anything incorrect here. I think a much more likely explanation is an error / bug in the GCS server (or some kind of throttling / load balancing component). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
