thinkharderdev commented on issue #4754: URL: https://github.com/apache/arrow-rs/issues/4754#issuecomment-1781389854
> Taking a step back, could your use-case encode the lifecycle details in the path of the object instead? No, ultimately it's not up to us (this was a solution in place before us and would be monumentally complex to change). > Having played around with this I'm unsure how to support this consistently, stores have different restrictions on what value are valid, and support for this across the stores is wildly inconsistent, even stores from the same provider... Why is this a problem? If a user adds incorrect metadata (values which are not allowed for whatever reason by the particular provider) then they get an error. It's no different than (for example) writing multi-part file to S3 in which case chunks need to be > 5.5MB (except for the last one). But the same limitation obviously wouldn't apply to local file systems. So at some level you have to know which provider you are using and what the individual semantics are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
