kdn36 commented on issue #726: URL: https://github.com/apache/arrow-rs-object-store/issues/726#issuecomment-4498220669
Thanks. While that would technically work, it sits in the infra management domain, which we want to avoid for a number of reasons (product boundaries mostly). We have a working version using `with_http_connector` with the caching sitting on our side (polars), but I ran into an integration challenge because the `ClientOptions` fields are private. Let me know if I missed something. In case there is no appetite to add the actual DNS caching inside object_store, these are the options I see: 1) Add a `pub fn client_builder()` to `impl ClientOptions` . This enables us to honor the client_options and add a custom connector. New PR (see also https://github.com/kdn36/arrow-rs-object-store/tree/feat_pub_client_builder for an unfinished working example). Downside: this may expose internals. 2) Extract the fields from `ClientOptions` individually and re-apply as needed in our `fn connect()`. This is fragile and adds maintenance overhead. 3) Add a `with_dns_resolver` acting on a `dyn Resolve` method to `AmazonS3Builder` (etc) to which we can pass our own DNS resolver (similar to the current `ShuffleResolver` (details to be worked out). Generalization of `RandomizeAddresses`. Option 3) sounds like the cleanest for both sides. We don't have to manage a custom `HttpConnector` and object_store does not have to expose internals. Appreciate your guidance. If it is not too much effort, I should be able to contribute to the chosen direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
