tustvold commented on issue #268: URL: https://github.com/apache/arrow-rs-object-store/issues/268#issuecomment-3023060903
Couple of points LimitStore is probably the wrong level to limit HTTP stores due to the internal backoff and retry machinery, and should use the HTTP layer It is unclear to me what the use-case for rate limiting LocalFileSystem is, and therefore what this should look like. The linked issue says on startup slatedb makes a huge number of get requests leading to high memory usage, it seems to me that it would be better to instead control the concurrency / memory usage of whatever process is initiating this, rather than constraining the IO. This would be a more reliable fix, whilst also likely more efficient. Ultimately throttling IO to avoid OOM seems fraught given it is the application that determines how long buffers live - e.g. throttling a `join_all` will just make it slower -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
