Cheappie commented on issue #2230: URL: https://github.com/apache/arrow-rs/issues/2230#issuecomment-1200144042
Sure, let me briefly explain what I am trying to achieve. I have implemented custom TableProvider in order to embed communication with external service that performs data-source indexing and produces binary stream with complete listing over data-source for given parameters that are part of SQL query expression. In addition to that I have implemented custom ObjectStore that provides just get operations. As you can see, what I would prefer most is to be able to provide binary blob instead of either Path or Url to get operation. But Url is closest thing that shares similarities with Path and is more flexible. Well in my case with existing ObjectStore trait, I will have to convert my binary listing to text and I will have to attach arguments to every Url unfortunately. > To give an example of where ambiguity would arise, if I call the list API with a URL instead of a prefix, what should the store do, should it ignore the extra arguments, should the returned Path also contain these arguments? Another layer of abstraction is my implementation of ObjectStore that embeds tiered storage, I don't know where would be better place to put that. Let me get to the point, I do need at least ObjectMeta struct in get operations of ObjectStore instead of just Path because I do need to invalidate tiered storage when last modified time doesn't match. I wouldn't mind if we could in addition to passing ObjectMeta to ObjectStore get ops, create new field within ObjectMeta like `custom_attributes: Option<Box<[u8]>> or Option<Arc<[u8]>>` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
