erickguan commented on code in PR #7818: URL: https://github.com/apache/opendal/pull/7818#discussion_r3467699309
########## core/core/src/docs/rfcs/7818_rename_if_not_exists.md: ########## @@ -0,0 +1,450 @@ +- Proposal Name: `rename_if_not_exists` +- Start Date: 2026-06-24 +- RFC PR: [apache/opendal#7818](https://github.com/apache/opendal/pull/7818) + +# Summary + +Extend rename with an `if_not_exists` option: + +```rust +op.rename_with("staging/file", "published/file") + .if_not_exists(true) + .await?; +``` + +The existing `rename` API keeps its overwrite semantics. When +`if_not_exists` is enabled, rename succeeds only if the destination does not +exist. A destination conflict returns `ConditionNotMatch` without modifying the +source or destination. + +# Motivation + +OpenDAL defines `rename` as an overwrite operation. Some applications also need +an atomic publish primitive: move a completed staging file into place only when +no other writer has already published that destination. + +A caller cannot implement this safely with `stat` followed by `rename`. Another +writer can create the destination after `stat` reports that it is absent but +before rename runs. A service configuration flag is also unsuitable because it +would make the meaning of the same `rename` call depend on backend construction +rather than an explicit call-site condition. + +OpenDAL already models the equivalent destination condition for write and copy +through options: + +```rust +op.write_with("path", content) + .if_not_exists(true) + .await?; + +op.copy_with("source", "target") + .if_not_exists(true) + .await?; +``` + +Rename should follow the same public API and error model. + +# Guide-level explanation + +Use `rename` when the destination may be replaced: + +```rust +use opendal::{Operator, Result}; + +async fn replace(op: Operator) -> Result<()> { + op.rename("staging/file", "published/file").await?; + Ok(()) +} +``` + +Use `rename_with(...).if_not_exists(true)` when an existing destination must be +preserved: + +```rust +use opendal::{ErrorKind, Operator, Result}; + +async fn publish(op: Operator) -> Result<()> { + match op + .rename_with("staging/file", "published/file") + .if_not_exists(true) + .await + { + Ok(()) => Ok(()), + Err(err) if err.kind() == ErrorKind::ConditionNotMatch => Err(err), + Err(err) => Err(err), + } +} +``` + +The conditional operation has the following outcomes: + +- If the destination does not exist, the source is renamed to the destination. +- If the destination exists, the operation returns `ConditionNotMatch` and + leaves both paths unchanged. +- If the service cannot enforce the destination condition atomically, the + operation returns `Unsupported`. +- If source and destination are the same path, the operation returns + `IsSameFile`, matching normal rename. + +Users can inspect `Capability::rename_with_if_not_exists` before enabling the +option. + +Blocking users configure the same condition through `RenameOptions`: + +```rust +use opendal::blocking; +use opendal::options::RenameOptions; +use opendal::Result; + +fn publish(op: blocking::Operator) -> Result<()> { + let mut options = RenameOptions::default(); + options.if_not_exists = true; + op.rename_options("staging/file", "published/file", options)?; + Ok(()) +} +``` + +# Reference-level explanation + +## Public API + +Add `RenameOptions`: + +```rust +#[derive(Debug, Clone, Default, Eq, PartialEq)] +pub struct RenameOptions { + pub if_not_exists: bool, +} +``` + +The asynchronous operator exposes: + +```rust +impl Operator { + pub async fn rename(&self, from: &str, to: &str) -> Result<()>; + + pub fn rename_with( + &self, + from: &str, + to: &str, + ) -> FutureRename<impl Future<Output = Result<()>>>; + + pub async fn rename_options( + &self, + from: &str, + to: &str, + options: impl Into<RenameOptions>, + ) -> Result<()>; +} +``` + +`rename` delegates to `rename_options` with default options. `FutureRename` +provides: + +```rust +impl<F: Future<Output = Result<()>>> FutureRename<F> { + pub fn if_not_exists(self, value: bool) -> Self; +} +``` + +The blocking operator follows existing blocking options APIs: + +```rust +impl blocking::Operator { + pub fn rename_options( + &self, + from: &str, + to: &str, + options: RenameOptions, + ) -> Result<()>; +} +``` + +No standalone `rename_if_not_exists` method is added. The options API matches +write and copy and leaves room for future composable rename conditions. + +The rename API follows the copy API at every layer: + +| Layer | Copy | Rename | +| --- | --- | --- | +| Default operation | `copy` | `rename` | +| Fluent options | `copy_with(...).if_not_exists(true)` | `rename_with(...).if_not_exists(true)` | +| Options struct | `CopyOptions` | `RenameOptions` | +| Explicit options call | `copy_options` | `rename_options` | +| Raw arguments | `OpCopy::if_not_exists()` | `OpRename::if_not_exists()` | +| Capability | `copy_with_if_not_exists` | `rename_with_if_not_exists` | + +This RFC does not add a public `copy_if_not_exists`-style standalone method, +because copy itself exposes the condition through its options APIs. + +## Service API + +`RenameOptions` converts into the raw `OpRename`: + +```rust +pub struct OpRename { + if_not_exists: bool, +} +``` + +The `Service::rename` signature does not change. Services inspect +`OpRename::if_not_exists()` to select their native overwrite or no-overwrite +operation. + +Add a capability field: + +```rust +pub struct Capability { + pub rename: bool, + pub rename_with_if_not_exists: bool, +} +``` + +`rename_with_if_not_exists` is meaningful only when `rename` is also supported. +The correctness check returns `Unsupported` before dispatch when the option is +enabled but the service does not advertise the capability. + +## Error semantics + +`if_not_exists` is a destination precondition, so a destination conflict maps +to `ConditionNotMatch`. This is consistent with existing write and copy +behavior. + +Backend errors caused by the native no-replace condition must be translated to +`ConditionNotMatch`, even when the backend or operating system reports a native +error such as `AlreadyExists`. + +Other errors retain their normal meanings. For example, a missing source +returns `NotFound`, a directory passed where a file is required returns the +corresponding directory error, and an unsupported condition returns +`Unsupported`. + +## Atomicity requirement + +A service must advertise `rename_with_if_not_exists` only when the destination +condition and rename are enforced by one native operation. Performing a +destination check followed by an overwriting rename has a time-of-check to +time-of-use race and does not satisfy the capability. + +When the destination condition fails: + +- The source remains at its original path. +- The destination remains unchanged. + +When the operation succeeds: + +- The source no longer exists at its original path. +- The destination contains the source data. + +This RFC does not require rename to be crash-atomic beyond the guarantees of the +underlying service. It requires the no-replace decision to be atomic with +respect to competing destination creation. + +## Service analysis + +### Audit scope + +At the time of this RFC, 19 services advertise `Capability::rename`. The table +below audits all of them, plus S3 as the representative object store without a +native rename operation. + +The "assessment" column describes whether the current backend primitive could +support `rename_with_if_not_exists`. It does not enable the capability. A +backend must still demonstrate that the destination check and move are one +atomic service operation and map destination conflicts to +`ConditionNotMatch`. + +| Service | Current rename implementation | Assessment for `if_not_exists` | +| --- | --- | --- | +| `aliyun_drive` | Deletes the destination, moves by file ID, then updates the name | Not suitable as implemented; overwrite is a multi-request workflow | +| `azdls` | Sends one Data Lake Storage rename request | Strong candidate: the API supports `If-None-Match: *` on the destination | +| `azfile` | Sends one Azure Files rename request with `x-ms-file-rename-replace-if-exists: true` | Strong candidate: set the native replace header to `false` | +| `compfs` | Calls `compio::fs::rename` | Same portability constraints as `fs` | +| `dbfs` | Sends one DBFS move request | Candidate, but atomicity must be verified; DBFS rejects an existing destination | +| `dropbox` | Sends one `files/move_v2` request | Candidate, but destination-conflict and atomicity guarantees must be verified | +| `fs` | Calls `tokio::fs::rename` | No portable no-replace primitive in the current implementation | +| `gdrive` | Trashes the destination, then patches source metadata | Not suitable; overwrite is a multi-request workflow and Drive permits duplicate names | +| `goosefs` | Stats and deletes the destination, then calls the native rename RPC | Candidate if the native conflict is atomic; the current overwrite path is multi-step | +| `hdfs_native` | Mutates the destination, then calls the client rename API with an overwrite flag | Strong candidate through the native `overwrite = false` path, after removing destination pre-mutation | +| `hdfs` | Deletes the destination for normal rename, then calls native no-overwrite rename | Supported by this RFC's initial implementation | +| `koofr` | Removes the destination, then sends one move request | Candidate only if the move endpoint atomically rejects conflicts; current overwrite is multi-step | +| `monoiofs` | Calls `monoio::fs::rename` | Same portability constraints as `fs` | +| `onedrive` | Sends a PATCH move request with conflict behavior set to `replace` | Candidate through conflict behavior `fail`, subject to OneDrive variant compatibility | +| `pcloud` | Sends one `renamefile` or `renamefolder` request | Requires service-specific conflict and atomicity verification | +| `sftp` | Calls the SFTP client's rename operation | Requires protocol/server-specific verification; overwrite behavior is not portable across servers | +| `upyun` | Sends one move request using `x-upyun-move-source` | Requires service-specific conflict and atomicity verification | +| `webdav` | Sends one `MOVE` request with `Overwrite: T` | Strong candidate: RFC 4918 defines `Overwrite: F` and a failed precondition | +| `yandex_disk` | Sends one move request with `overwrite=true` | Candidate through `overwrite=false`, subject to atomicity and error-mapping verification | +| `s3` | Does not advertise rename; supports copy and delete separately | Not suitable: conditional copy followed by delete is not an atomic rename | + +This audit also reveals that several existing backends implement OpenDAL's +overwrite contract by deleting the destination before a native move. That +behavior is inherently multi-step, but it does not prevent the same backend +from supporting conditional rename when its native move atomically rejects an +existing destination. The conditional capability must be based on the native +operation, not on a `stat` or delete preflight. + +### HDFS + +HDFS exposes rename options through `Options.Rename`. libhdfs `hdfsRename` +invokes rename with `Options.Rename.NONE`, which rejects an existing +destination. The HDFS service can therefore advertise +`rename_with_if_not_exists` and use that native operation directly. + +Normal OpenDAL rename continues to provide overwrite behavior. Where the +libhdfs operation does not overwrite, the HDFS service removes an existing file +destination before invoking native rename. + +The HDFS implementation must translate both an existing destination discovered +before the call and a native conflict reported by the final rename into +`ConditionNotMatch`. The final native operation remains authoritative so a +concurrent creator cannot be overwritten. + +### FS + +The FS service currently uses `tokio::fs::rename`. Normal rename behavior and +destination replacement differ across operating systems, and this API does not +offer a portable no-replace option. + +The FS service must not advertise `rename_with_if_not_exists` with a +`metadata`-then-`rename` implementation. A future implementation may use native +primitives such as Linux `renameat2(RENAME_NOREPLACE)` and equivalent supported +operations on other platforms, with a clearly defined portability policy. + +Any existing cross-platform differences in default FS overwrite behavior are +outside this RFC and should be handled separately. + +`compfs` and `monoiofs` wrap the same operating-system rename model through +different runtimes, so they have the same portability constraint. SFTP also +requires caution because the effective rename behavior depends on the protocol +operation and server implementation. + +### Native conditional move services + +Several services expose a destination policy in the same server-side move +request: + +- Azure Data Lake Storage accepts `If-None-Match: *` for rename. +- Azure Files exposes `x-ms-file-rename-replace-if-exists`. +- WebDAV defines the `Overwrite` header for `MOVE`; `F` requires the server to + reject an existing destination. +- Yandex Disk exposes an `overwrite` query parameter. +- OneDrive exposes conflict behavior, although support differs between service + variants and must be verified before advertising the capability. + +These services are candidates for follow-up implementations. They are not +enabled by this RFC's initial HDFS change. Each follow-up must include behavior +coverage and service-specific error translation. + +### Multi-step overwrite services + +`aliyun_drive`, `gdrive`, `goosefs`, and `koofr` currently delete or trash an Review Comment: @hfutatzhanghb Is zhb your handle on OpenDAL discord? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
