waynr commented on PR #7170: URL: https://github.com/apache/arrow-rs/pull/7170#issuecomment-2675261657
> You could add DataFusion's entire Extensions bundle here, whilst you would have two indirections, it would be the least magical mechanism for doing this. Do you mean that instead of the `trait Extension` approach shown in this PR we do exactly as is done for extensions in datafusion, ie just use a `HashMap<TypeId, Arc<dyn Any + Send + Sync + 'static>>`? > That being said, I am increasingly of the opinion the ObjectStore trait is being overloaded for something it fundamentally is not designed to be, and I would be interested in your thoughts on #7171. Yeah I'm taking a look at that now, I'll see if I can think of any useful feedback for you there. > It seems really odd to me that we need to extend the ObjectStore trait, a generic interface to object stores, to support propagating DF context through to an IOx subsystem. This screams to me of a missing / incorrect layer of abstraction. I wonder if instead IOx could be using a less generic trait, e.g. overriding AsyncFileReaderFactory, or similar? I'm still pretty new to datafusion ecosystem but it seems to me like using it the way influxdb is more or less requires passing in some kind of `Arc<dyn ObjectStore>` so I don't yet see how overriding `AsyncFileReaderFactory` would be a viable alternative here -- unless there is some logic in the datafusion execution path I missed that would prefer that over an object store. Or are you suggesting a refactor of datafusion abstractions to support that approach? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
