thinkharderdev commented on issue #4754:
URL: https://github.com/apache/arrow-rs/issues/4754#issuecomment-1781580786
> > It's no different than (for example) writing multi-part file to S3 in
which case chunks need to be > 5.5MB (except for the last one). But the same
limitation obviously wouldn't apply to local file systems. So at some level you
have to know which provider you are using and what the individual semantics are.
>
> Because in general we try to hide these incompatibilities from you, you
can't write to funky paths, the chunking for multipart upload is done for you,
etc... We could add TagSets to the crate, and I have a mostly complete PR that
does this, but it just seems strange to add something to the ObjectStore trait
that is supported by only 1 and a half stores...
Right, and I think it's a good idea to try and hide the incompatibilities,
but if the only way to do that is not add the functionality at all then it may
be better to just expose the incompatibilities and let user's deal with it. I
guess the "proper" way to do this would be through traits. You could have the
base `ObjectStore` trait expose the minimal API surface area that every
provider can implement. And then have other traits for stuff not supported by
all providers (`ObjectAppend`, `ObjectMetadata`, etc). This is a little awkward
for upstream projects like DataFusion which tend to pass around `Arc<dyn
ObjectStore>` but maybe this can be handled dynamically as well. So maybe
something like
```
trait ObjectAppend: ObjectStore {
async fn append(&self, location: &Path, bytes: Bytes) -> Result<()>;
}
trait ObjectStore {
.. regular methods
fn as_append(&self) -> Option<Arc<dyn ObjectAppend>>;
// or once RPIT lands on stable
fn as_append(&self) -> Option<&impl ObjectAppend>;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
