flaneur2020 commented on issue #7251: URL: https://github.com/apache/arrow-rs/issues/7251#issuecomment-2708718945
> I believe this is similar to the OpenDAL [concept of layers](https://docs.rs/opendal/latest/opendal/#compose-layers) but @Xuanwo please correct me if I am wrong i suppose this might be more resemble with the "Operator" concept in OpenDAL, which can help handling the chunking & concurrency parameter in a builder pattern like [this](https://opendal.apache.org/docs/rust/opendal/struct.Reader.html#concurrent-read): ``` let s = op .reader_with("hello.txt") .concurrent(8) .chunk(256) .await? .into_stream(1024..2048) .await?; ``` `Layer` is a bit lower level than this imo, we can wrap caches to s3 operations like `read`, `write`, while chunking & concurrency is handled in the `Operator` side 🤔. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
