This is an automated email from the ASF dual-hosted git repository.
xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-opendal.git
The following commit(s) were added to refs/heads/main by this push:
new 470fc2fe39 RFC-3911: Deleter API (#3911)
470fc2fe39 is described below
commit 470fc2fe397c8e8973aa3bc190969d07f1bee110
Author: Xuanwo <[email protected]>
AuthorDate: Sat Jan 6 11:43:09 2024 +0800
RFC-3911: Deleter API (#3911)
* Save work
Signed-off-by: Xuanwo <[email protected]>
* Add deleter_api
Signed-off-by: Xuanwo <[email protected]>
* Assign number
Signed-off-by: Xuanwo <[email protected]>
* Polish
Signed-off-by: Xuanwo <[email protected]>
* Add tracking issues
Signed-off-by: Xuanwo <[email protected]>
* Reduce confusion
Signed-off-by: Xuanwo <[email protected]>
* Fix typo
Signed-off-by: Xuanwo <[email protected]>
---------
Signed-off-by: Xuanwo <[email protected]>
---
core/src/docs/rfcs/3911_deleter_api.md | 172 +++++++++++++++++++++++++++++++++
core/src/docs/rfcs/mod.rs | 6 ++
2 files changed, 178 insertions(+)
diff --git a/core/src/docs/rfcs/3911_deleter_api.md
b/core/src/docs/rfcs/3911_deleter_api.md
new file mode 100644
index 0000000000..b87636b0d0
--- /dev/null
+++ b/core/src/docs/rfcs/3911_deleter_api.md
@@ -0,0 +1,172 @@
+- Proposal Name: `deleter_api`
+- Start Date: 2024-01-04
+- RFC PR:
[apache/incubator-opendal#3911](https://github.com/apache/incubator-opendal/pull/3911)
+- Tracking Issue:
[apache/incubator-opendal#3922](https://github.com/apache/incubator-opendal/issues/3922)
+
+# Summary
+
+Introduce the `Deleter` API to enhance batch and recursive deletion
capabilities.
+
+# Motivation
+
+All OpenDAL's public API follow the same design:
+
+- `read`: Execute a read operation.
+- `read_with`: Execute a read operation with additional options, like range
and if_match.
+- `reader`: Create a reader for streaming data, enabling flexible access.
+- `reader_with`: Create a reader with advanced options.
+
+However, `delete` operations vary. OpenDAL offers several methods for file
deletion:
+
+- `delete`: Delete a single file or an empty dir.
+- `remove`: Remove a list of files.
+- `remove_via`: Remove files produced by a stream.
+- `remove_all`: Remove all files under a path.
+
+This design is not consistent with the other APIs, and it is not easy to use.
+
+So I propose `Deleter` to address them all at once.
+
+# Guide-level explanation
+
+The following new API will be added to `Operator`:
+
+```diff
+impl Operator {
+ pub async fn detele(&self, path: &str) -> Result<()>;
++ pub async fn delete_with(&self, path: &str) -> FutureDelete;
+
++ pub async fn deleter(&self) -> Result<Deleter>;
++ pub async fn deleter_with(&self) -> FutureDeleter;
+}
+```
+
+- `delete` is the existing API, which deletes a single file or an empty dir.
+- `delete_with` is an extension of the existing `delete` API, which supports
additional options, such as `recursive`.
+- `deleter` is a new API that returns a `Deleter` instance.
+- `deleter_with` is an extension of the existing `deleter` API, which supports
additional options, such as `recursive`.
+
+The following new options will be available for `delete_with` and
`deleter_with`:
+
+- `recursive`: Enable recursive deletion.
+- `concurrent`: How many delete tasks can be performed concurrently?
+- `buffer`: How many files can be buffered for send in a single batch?
+
+Users can delete a file recursively in this way:
+
+```rust
+let _ = op.delete_with("path/to/file").recursive(true).await?;
+```
+
+Users can delete multiple files in this way:
+
+
+```rust
+let deleter = op.deleter().await?;
+
+// Add a single file to the deleter.
+deleter.delete(path).await?;
+
+// Add a stream of files to the deleter.
+deleter.delete_all(&mut lister).await?;
+
+// Close deleter, make sure all input files are deleted.
+deleter.close().await?;
+```
+
+`Deleter` also implements
[`Sink`](https://docs.rs/futures/latest/futures/sink/trait.Sink.html), so all
the methods of `Sink` are available for `Deleter`. For example, users can use
[`forward`](https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.forward)
to forward a stream of files to `Deleter`:
+
+```rust
+// Init a deleter to start batch delete tasks.
+let deleter = op.deleter().await?;
+// List all files that ends with tmp
+let lister = op.lister(path).await?
+ .filter(|x|future::ready(x.ends_with(".tmp")));
+
+// Forward all paths into deleter.
+lister.forward(deleter).await?;
+
+// Send all from a stream into deleter.
+deleter.send_all(&mut lister).await?;
+
+// Close the deleter.
+deleter.close().await?;
+```
+
+Users can control the behavior of `Deleter` by setting the options:
+
+```rust
+let deleter = op.deleter()
+ // Allow up to 8 concurrent delete tasks, default to 1.
+ .concurrent(8)
+ // Configure the buffer size to 1000, default value provided by services.
+ .buffer(1000)
+ .await?;
+
+// Add a single file to the deleter.
+deleter.delete(path).await?;
+
+// Add a stream of files to the deleter.
+deleter.delete_all(&mut lister).await?;
+
+// Close deleter, make sure all input files are deleted.
+deleter.close().await?;
+```
+
+In response to `Deleter` API, we will remove APIs like `remove`, `remove_via`
and `remove_all`.
+
+- `remove` and `remove_via` could be replaced by `Deleter` directly.
+- `remove_all` could be replaced by `delete_with(path).recursive(true)`.
+
+# Reference-level explanation
+
+To provide those public APIs, we will add a new associated type in `Accessor`:
+
+```rust
+trait Accessor {
+ ...
+
+ type Deleter = oio::Delete;
+ type BlockingDeleter = oio::BlockingDelete;
+}
+```
+
+And the `delete` API will be changed to return a `oio::Delete` instead:
+
+```diff
+trait Accessor {
+- async fn delete(&self, path: &str, args: OpDelete) -> Result<RpDelete>;
++ async fn delete(&self, path: &str, args: OpDelete) -> Result<(RpDelete,
Self::Deleter)>;
+}
+```
+
+Along with this change, we will remove the `batch` API from `Accessor`:
+
+```rust
+trait Accessor {
+- async fn batch(&self, args: OpBatch) -> Result<RpBatch>;
+}
+```
+
+# Drawbacks
+
+- Big breaking changes.
+
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Add API that accepts `IntoIterator`
+
+It's possible to add a new API that accepts `IntoIterator` so users can input
`Vec<String>` or `Iter<String>` into `Deleter`.
diff --git a/core/src/docs/rfcs/mod.rs b/core/src/docs/rfcs/mod.rs
index 57bc6824d4..1d665ef273 100644
--- a/core/src/docs/rfcs/mod.rs
+++ b/core/src/docs/rfcs/mod.rs
@@ -217,8 +217,14 @@ pub mod rfc_3526_list_recursive {}
#[doc = include_str!("3574_concurrent_stat_in_list.md")]
pub mod rfc_3574_concurrent_stat_in_list {}
+/// Buffered Reader
#[doc = include_str!("3734_buffered_reader.md")]
pub mod rfc_3734_buffered_reader {}
+/// Concurrent Writer
#[doc = include_str!("3898_concurrent_writer.md")]
pub mod rfc_3898_concurrent_writer {}
+
+/// Deleter API
+#[doc = include_str!("3911_deleter_api.md")]
+pub mod rfc_3911_deleter_api {}
