winding-lines commented on PR #3429: URL: https://github.com/apache/arrow-rs/pull/3429#issuecomment-1369306328
I support any approach that we can use across multiple clients, specially if it has the support of the object-storage devs. On Mon, Jan 2, 2023 at 5:06 PM Robert Pack ***@***.***> wrote: > As @winding-lines <https://github.com/winding-lines> mentioned, in > delta-rs we are faced with similar questions and also want to support > configration via python that resemble pyarrow / fsspec confgurations, as > well as configration within datafusion queries > <https://github.com/delta-io/delta-rs/pull/1043>, similar to what you can > do in spark. > > The way we went there is so work with a property bag potentially augmented > by configuration from the environment. As it turns out, I very recently > experimented with an iteration on the higher level builder > <https://github.com/roeap/object-store-python/blob/main/object-store/src/builder.rs> > that handles creating differnt kinds of object stores, which I am planning > to move to delta.rs. > > That said, I definately see the usecase, but also share @tustvold > <https://github.com/tustvold>'s concerns. My personal preference (and I > may be biased :)) would be to handle these use cases, by just sending > property bags around, that allow for configuration based on > "well-known-keys" that may also be in the environment. Not entirely sure > though if the implementation linked above is the way to go, or if it is > maybe going a bit too far in what alternatives are possible. > > Mainly complexities arise when supporting multiple aliases for the same > configuration value and handling connections to various stores in the same > environment. > > If there is general interest already I'd be happy to open a PR for a more > concrete discussion - if that would also meet your needs @winding-lines > <https://github.com/winding-lines>. > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/arrow-rs/pull/3429#issuecomment-1369303568>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAAJYPQK7K7SMKMOMXJPARLWQN3SNANCNFSM6AAAAAATO4ISEQ> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
