Cheappie commented on issue #2230: URL: https://github.com/apache/arrow-rs/issues/2230#issuecomment-1200452349
I have taken a look into datafusion codebase and from what I have seen baking my own ExecutionPlan to replace ObjectStore would require quite large effort. Mainly rewriting ParquetExec, FileScanConfig, and some other components along the way that are bound to ObjectStore. I feel like anyway we go, at the end there will be similar or yet another abstraction of `ObjectStore` in DataFusion. That will have more relaxed semantics than the one that is being moved into arrow. I like the idea of request preconditions, but It won't be sufficient for me. I've remembered that I would need to identify who has requested object from ObjectStore. It's good example of why It would be nice to have ability to pass dynamic attributes through something like `Box<[u8]>`. > > What do you think about modifying ObjectStore get operations to replace Path with ObjectMeta and adding custom_attributes: Option<Box<[u8]>> to ObjectMeta ? > > I think this is at odds with the objectives of the crate to provide a consistent abstraction across object stores, and so I would be extremely reticent to change the API in this way Could you elaborate ? What is wrong in providing ObjectMeta to get operation ? I mean that is the output of list operation. Or are you worried about custom attributes field ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
