Xuanwo commented on code in PR #3574: URL: https://github.com/apache/incubator-opendal/pull/3574#discussion_r1393579596
########## core/src/docs/rfcs/3574_concurrent_stat_in_list.md: ########## @@ -0,0 +1,81 @@ +- Proposal Name: `concurrent_stat_in_list` +- Start Date: 2023-11-13 +- RFC PR: [apache/incubator-opendal#3574](https://github.com/apache/incubator-opendal/pull/3574) +- Tracking Issue: [apache/incubator-opendal#3575](https://github.com/apache/incubator-opendal/issues/3575) + +# Summary + +Add concurrent stat in list operation. + +# Motivation + +[RFC-2779](https://github.com/apache/incubator-opendal/pull/2779) allows user to list with metakey. +However, the stat inside list could make the list process much slower. We should allow concurrent stat during list so that stat could be sent concurrently. + + +# Guide-level explanation + +For users who want to concurrently run statistics in list operations, they will call the new API `concurrent`. The `concurrent` function will take a number as input, and this number will represent the maximum concurrent stat handlers. + +The default behavior remains unchanged, so users using `op.list_with()` are not affected. And this implementation should be zero cost to users who don't want to do concurrent stat. + +```rust +op.lister_with(path).metakey(Metakey::ContentLength).concurrent(10).await? +``` + + +# Reference-level explanation + +When `concurrent` is set and `list_with` is called, the list operation will be split into two parts: list and stat. + +The list part will iterate through the entries inside the buffer, and if its `metakey` is unknown, it will send a stat request to the storage service. + +We will add a new field `concurrent` to `OpList`. The type of `concurrent` is `Option<u32>`. If `concurrent` is `None`, it means the default behavior. If `concurrent` is `Some(n)`, it means the maximum concurrent stat handlers are `n`. + +Then we could use a sized `VecDeque` to limit the maximum concurrent stat handlers. Additionally, we could use handlers `JoinHandle<T>` inside `VecDeque` to spawn and queue the stat tasks. +While iterating through the entries, we should check if the `metakey` is unknown and if the `VecDeque` is full. If the `metakey` is unknown and the `VecDeque` is full, we should wait and join the handle once it’s finished, since we need to keep the entry order. + +If the metakey is unknown and the handlers are full, we should break the loop and wait for the spawned tasks inside handlers to finish. After the spawned tasks finish, we should iterate through the handlers and return the result. + +If the metakey is known, we should check if the handlers are empty. If true, return the result immediately; otherwise, we should wait for the spawned tasks to finish. + +# Drawbacks + +1. More memory usage +2. More complex code +3. More complex testing + +# Rationale and alternatives + +## Why not `VecDeque<BoxFuture<'static, X>>`? + +To maintain the order of returned entries, we need to pre-run future entries before returning the current one to address the slowness issue. +Although we could use `VecDeque<BoxFuture<'static, X>>` to store the spawned tasks, +using it here would prevent us from executing the async block concurrently when we only have one `cx: &mut Context<'_>`. + +## Do we need `Semaphore`? + +No, we can control the concurrent number by limiting the length of the `VecDeque`. +Using a `semaphore` will introduce more cost and memory. + +## Why not using `JoinSet`? + +`JoinSet` requires mutability to spawn or join the next task, and `tokio::spawn()` requires the async block to be `'static`. Review Comment: The mainly reason for not using `JoinSet` is it can't keep the order. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
