zeroshade commented on issue #44135: URL: https://github.com/apache/arrow/issues/44135#issuecomment-2356034004
> one downside we see is that everytime it makes an IO call to read from S3 .. but data will be read fresh That's why i suggested potentially having an LRU cache or otherwise to cache *some* datasets (not all!) somewhere (Redis, etcd, local memory, whatever) so that your most frequently requested data could avoid the IO call to s3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
