roeap commented on issue #4047: URL: https://github.com/apache/arrow-rs/issues/4047#issuecomment-1503929734
From my perspective, centralising this logic makes a lot of sense. We actually have some preparatory work already done within object store. Specifically, extracting relevant information from the URLs is available on the builders via respective `parse_url` methods. I guess it makes sense to reuse that logic somehow. One thing we discussed back then was the option to go as far as moving the `ObjectStoreFactory` (not sure what its actually called right now 😆 ) trait from datafusion into the object store crate. But not sure what my stance on that would be today. I guess a simple "global" `parse_url` function like @tustvold suggest would also be my preference today. There is one more kind of related thing I recently looked into, but haven't converged on a good solution yet. Essentially using `from_env` may lead to a situation where a complete credential specified in the storage options will not be chosen, b/c there is sufficient information available in the environment to in principle authorise. On the other hand, creating an ObjectStore without initializing options from the environment will always succeed (at least for AWS and Azure) since there are config-free options available via the metadata endpoint. My current thinking was to check storage options if they contain a full credential (this would have to be build) and if not, then initialise from the environment. In azure the default credential allows for explicitly configuring to omit certain options as an additional guard against picking the wrong credential, which may be an option as well. All in all though I think this is a gerat addition to object store. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
