suyanhanx commented on code in PR #3574: URL: https://github.com/apache/incubator-opendal/pull/3574#discussion_r1393845110
########## core/src/docs/rfcs/3574_concurrent_stat_in_list.md: ########## @@ -0,0 +1,80 @@ +- Proposal Name: `concurrent_stat_in_list` +- Start Date: 2023-11-13 +- RFC PR: [apache/incubator-opendal#3574](https://github.com/apache/incubator-opendal/pull/3574) +- Tracking Issue: [apache/incubator-opendal#3575](https://github.com/apache/incubator-opendal/issues/3575) + +# Summary + +Add concurrent stat in list operation. + +# Motivation + +[RFC-2779](https://github.com/apache/incubator-opendal/pull/2779) allows user to list with metakey. +However, the stat inside list could make the list process much slower. We should allow concurrent stat during list so that stat could be sent concurrently. + + +# Guide-level explanation + +For users who want to concurrently run statistics in list operations, they will call the new API `concurrent`. The `concurrent` function will take a number as input, and this number will represent the maximum concurrent stat handlers. + +The default behavior remains unchanged, so users using `op.list_with()` are not affected. And this implementation should be zero cost to users who don't want to do concurrent stat. + +```rust +op.lister_with(path).metakey(Metakey::ContentLength).concurrent(10).await? +``` + + +# Reference-level explanation + +When `concurrent` is set and `list_with` is called, the list operation will be split into two parts: list and stat. + +The list part will iterate through the entries inside the buffer, and if its `metakey` is unknown, it will send a stat request to the storage service. + +We will add a new field `concurrent` to `OpList`. The type of `concurrent` is `Option<u32>`. If `concurrent` is `None`, it means the default behavior. If `concurrent` is `Some(n)`, it means the maximum concurrent stat handlers are `n`. Review Comment: Is it possible for the `n` here to be provided by an implementation of the service as a default value? So in some cases, like Some(0), the value provided by the service is used. (Of course, 0 could also mean unlimited, and this can be discussed further.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
