thinkharderdev commented on issue #4754: URL: https://github.com/apache/arrow-rs/issues/4754#issuecomment-1772987316
> Are you referring to https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html or some custom system? Both. The data is in customer buckets and we add tags so they can manage their own retention. How they do that is up to them, we just provide the tags. Currently we only need to write them. We can obviously work around that (and will in the immediate term) without involving the `ObjectStore` trait but it would be nice if we didn't have to as associating metadata with objects is used in a lot of applications. > Apart from this crate goes to great lengths to try to provide an API that is consistent across providers... 😅 Yeah, agreed but the APIs are what they are :). So we can either provide a consistent API which always works the same across providers by always doing additional API calls to grab metadata/tags (which seems like a bad idea). Or we can make the semantics around metadata depend on the provider. Or of course we can do neither and just say that if we can't provide consistent semantics because of provider API differences then it's not going to be exposed in the `ObjectStore` interface. But IMO that ship has already sailed. We have `ObjectStore::append` even though S3 and GCS don't support append operations at all and on Azure you can only append to objects that were created as append blobs to begin with. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
