Xuanwo commented on code in PR #3898: URL: https://github.com/apache/incubator-opendal/pull/3898#discussion_r1439602398
########## core/src/docs/rfcs/3898_concurrent_writer.md: ########## @@ -0,0 +1,48 @@ +- Proposal Name: `concurrent_writer` +- Start Date: 2024-01-02 +- RFC PR: [apache/incubator-opendal#3898](https://github.com/apache/incubator-opendal/pull/3898) +- Tracking Issue: [apache/incubator-opendal#3899](https://github.com/apache/incubator-opendal/issues/3899) + +# Summary + +Add concurrent write in `MultipartUploadWriter`. + +# Motivation + +The [object_writer](./1420_object_writer.md) introduces the `ObjectWriter` multipart upload support. However, the multiple parts are currently uploaded serially without fully leveraging the potential for improved throughput through concurrent uploads. We should support the upload of multiple parts concurrently. + +# Guide-level explanation + +For users who want to concurrent writer, they will call the new API `concurrent`. And the default behavior remains unchanged, so users using `op.writer_with()` are not affected. The `concurrent` function will take a number as input, and this number will represent the maximum concurrent write task amount the writer can perform. + +```rust +op.writer_with(path).concurrent(8).await +``` + +# Reference-level explanation + +This feature will be implemented in the `MultipartUploadWriter`, which will utilize a `ConcurrentFutures<WriteTask>` as a task queue to store concurrent write tasks. Review Comment: Although not all services may support this feature, we should consider them all. As we design features for OpenDAL, our aim is to ensure consistent behavior across services. For examples: - What will happen if services support concurrent write? - How will they support this feature? - How will services with `MultipartUploadWriter` implement? - How will services with `RangeWriter` implement? - What will happen if services doesn't support this feature? - Will users got an error? - Or will we fallback to normal write? - Is it possible in the future? - For example, we know `fs` can't support concurrent write. But what if we use `write_at` instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
