roeap commented on PR #3429: URL: https://github.com/apache/arrow-rs/pull/3429#issuecomment-1369303568
As @winding-lines mentioned, in delta-rs we are faced with similar questions and also want to support configration via python that resemble pyarrow / fsspec confgurations, as well as configration within [datafusion queries](https://github.com/delta-io/delta-rs/pull/1043), similar to what you can do in spark. The way we went there is so work with a property bag potentially augmented by configuration from the environment. As it turns out, I very recently experimented with an iteration on the [higher level builder](https://github.com/roeap/object-store-python/blob/main/object-store/src/builder.rs) that handles creating differnt kinds of object stores, which I am planning to move to delta.rs. That said, I definately see the usecase, but also share @tustvold's concerns. My personal preference (and I may be biased :)) would be to handle these use cases, by just sending property bags around, that allow for configuration based on "well-known-keys" that may also be in the environment. Not entirely sure though if the implementation linked above is the way to go, or if it is maybe going a bit too far in what alternatives are possible. Mainly complexities arise when supporting multiple aliases for the same configuration value and handling connections to various stores in the same environment. If there is general interest already I'd be happy to open a PR for a more concrete discussion - if that would also meet your needs @winding-lines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
