This is an automated email from the ASF dual-hosted git repository.
xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-opendal.git
The following commit(s) were added to refs/heads/main by this push:
new ddad905d RFC-2133: Introduce Append API (#2133)
ddad905d is described below
commit ddad905d581151cbc025ea971e1db7d939e8d0f8
Author: SSpirits <[email protected]>
AuthorDate: Sat Apr 29 17:43:27 2023 +0800
RFC-2133: Introduce Append API (#2133)
* rfc: introduce append api
* rfc: assign rfc serial number
* rfc: add `Append` trait
* rfc: focus this rfc on adding append operation
* rfc: add tracking issue
---
core/src/docs/rfcs/2133_append_api.md | 88 +++++++++++++++++++++++++++++++++++
core/src/docs/rfcs/mod.rs | 3 ++
2 files changed, 91 insertions(+)
diff --git a/core/src/docs/rfcs/2133_append_api.md
b/core/src/docs/rfcs/2133_append_api.md
new file mode 100644
index 00000000..41583c23
--- /dev/null
+++ b/core/src/docs/rfcs/2133_append_api.md
@@ -0,0 +1,88 @@
+- Proposal Name: `append_api`
+- Start Date: 2023-04-26
+- RFC PR:
[apache/incubator-opendal#2133](https://github.com/apache/incubator-opendal/pull/2133)
+- Tracking Issue:
[apache/incubator-opendal#2163](https://github.com/apache/incubator-opendal/issues/2163)
+
+# Summary
+
+Introduce append operations for OpenDAL which allow users to add data to a
file.
+
+# Motivation
+
+OpenDAL has the write operation used to create a file and upload in parts.
This is implemented based on multipart API. However, current approach has some
limitations:
+
+- Data could be lost and not readable before w.close() returned Ok(())
+- File can't be appended again after w.close() returned Ok(())
+
+To address these issues, I propose adding an append operation. Users can
create an appender that provides a reentrant append operation. Each append
operation will add data to the end of the file, which can be read immediately
after the operation.
+
+# Guide-level explanation
+
+The files created by the append operation can be appended via append API.
+
+```rust
+async fn append_test(op: Operation) -> Result<()> {
+ // create writer
+ let append = op.append("path_to_file").await?;
+
+ let bs = read_from_file();
+ append.append(bs).await?;
+
+ let bs = read_from_another_file();
+ append.append(bs).await?;
+
+ // close the file
+ append.close().await?;
+}
+```
+
+Difference between the write and append operation:
+
+- write: Always create a new file, not readable until close.
+- append: Can append existing appendable file, readable after append return.
+
+# Reference-level explanation
+
+For underlay API, we will make these changes in Accessor:
+
+```rust
+trait Accessor {
+ type Appender: oio::Append;
+
+ async fn append(&self, path: &str, args: OpAppend) -> Result<Self::Append>;
+}
+```
+
+To implement this feature, we need to add a new API `append` into
`oio::Append`.
+
+```rust
+#[async_trait]
+pub trait Append: Unpin + Send + Sync {
+ /// Append data to the end of file.
+ /// Users will call `append` multiple times. Please make sure `append` is
safe to re-enter.
+ async fn append(&mut self, bs: Bytes) -> Result<()>;
+
+ /// Seal the file to mark it as unmodifiable.
+ async fn close(&mut self) -> Result<()>;
+}
+```
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+We can use append API to implement for services that natively support append,
such as [Azure
blob](https://learn.microsoft.com/en-us/rest/api/storageservices/append-block?tabs=azure-ad)
and [Alibaba cloud
OSS](https://www.alibabacloud.com/help/en/object-storage-service/latest/appendobject).
This will improve the performance and reliability of append operation.
diff --git a/core/src/docs/rfcs/mod.rs b/core/src/docs/rfcs/mod.rs
index 7d6aed62..a27593b0 100644
--- a/core/src/docs/rfcs/mod.rs
+++ b/core/src/docs/rfcs/mod.rs
@@ -136,3 +136,6 @@ pub mod rfc_1735_operation_extension {}
#[doc = include_str!("2083_writer_sink_api.md")]
pub mod rfc_2083_writer_sink_api {}
+
+#[doc = include_str!("2133_append_api.md")]
+pub mod rfc_2133_append_api {}
