tisonkun commented on code in PR #5495: URL: https://github.com/apache/opendal/pull/5495#discussion_r1900592283
########## core/src/docs/rfcs/5495_list_with_deleted.md: ########## @@ -0,0 +1,81 @@ +- Proposal Name: `list_with_deleted` +- Start Date: 2025-01-02 +- RFC PR: [apache/opendal#5495](https://github.com/apache/opendal/pull/0000) +- Tracking Issue: [apache/opendal#5496](https://github.com/apache/opendal/issues/5496) + +# Summary + +Add `list_with(path).deleted(true)` to enable users to list deleted files from storage services. + +# Motivation + +OpenDAL is currently working on adding support for file versions, allowing users to read, list, and delete them. + +```rust +// Read given version +op.read_with(path).version(version_id).await; +// Fetch the metadata of given version. +op.stat_with(path).version(version_id).await; +// Delete the given version. +op.delete_with(path).version(version_id).await; +// List the path's versions. +op.list_with(path).versions().await; +``` + +However, to implement the complete data recovery workflow, we should also include support for recovering deleted files from storage services. This feature is referred to as `DeleteMarker` in S3 and `Soft Deleted` in Azure Blob Storage or Google Cloud Storage. Users can utilize these deleted files (or versions) to restore files that may have been accidentally deleted. + +# Guide-level explanation + +I suggest adding `list_with(path).deleted(true)` to allow users to list deleted files from storage services. + +```rust +let entries = op.list_with(path).deleted(true).await; +``` + +Please note that `deleted` here means "including deleted files" rather than "only deleted files." Therefore, `list_with(path).deleted(true)` will list both current files and deleted ones. + +At the same time, we will add an `is_deleted` field to the `Metadata` struct to indicate whether the file has been deleted. Together with the existing `is_current` field, we will have the following matrix: + +| `is_current` | `is_deleted` | Description | +|---------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Some(true)` | `false` | **The metadata's associated version is the latest, current version.** This is the normal state, indicating that this version is the most up-to-date and accessible version. | +| `Some(true)` | `true` | **The metadata's associated version is the latest, deleted version (Latest Delete Marker or Soft Deleted).** This is particularly important in object storage systems like S3. It signifies that this version is the **most recent delete marker**, indicating the object has been deleted. Subsequent GET requests will return 404 errors unless a specific version ID is provided. | +| `Some(false)` | `false` | **The metadata's associated version is neither the latest version nor deleted.** This indicates that this version is a previous version, still accessible by specifying its version ID. | +| `Some(false)` | `true` | **The metadata's associated version is not the latest version and is deleted.** This represents a historical version that has been marked for deletion. Users will need to specify the version ID to access it, and accessing it may be subject to specific delete marker behavior (e.g., in S3, it might not return actual data but a specific delete marker response). | +| `None` | `false` | **The metadata's associated file is not deleted, but its version status is either unknown or it is not the latest version.** This likely indicates that versioning is not enabled for this file, or versioning information is unavailable. | +| `None` | `true` | **The metadata's associated file is deleted, but its version status is either unknown or it is not the latest version.** This typically means the file was deleted without versioning enabled, or its versioning information is unavailable. This may represent an actual data deletion operation rather than an S3 delete marker. | Review Comment: Seems deleted only relevant if versioning are enabled? If so, then these two variants seems not applicible. ########## core/src/docs/rfcs/5495_list_with_deleted.md: ########## @@ -0,0 +1,81 @@ +- Proposal Name: `list_with_deleted` +- Start Date: 2025-01-02 +- RFC PR: [apache/opendal#5495](https://github.com/apache/opendal/pull/0000) +- Tracking Issue: [apache/opendal#5496](https://github.com/apache/opendal/issues/5496) + +# Summary + +Add `list_with(path).deleted(true)` to enable users to list deleted files from storage services. + +# Motivation + +OpenDAL is currently working on adding support for file versions, allowing users to read, list, and delete them. + +```rust +// Read given version +op.read_with(path).version(version_id).await; +// Fetch the metadata of given version. +op.stat_with(path).version(version_id).await; +// Delete the given version. +op.delete_with(path).version(version_id).await; +// List the path's versions. +op.list_with(path).versions().await; +``` + +However, to implement the complete data recovery workflow, we should also include support for recovering deleted files from storage services. This feature is referred to as `DeleteMarker` in S3 and `Soft Deleted` in Azure Blob Storage or Google Cloud Storage. Users can utilize these deleted files (or versions) to restore files that may have been accidentally deleted. + +# Guide-level explanation + +I suggest adding `list_with(path).deleted(true)` to allow users to list deleted files from storage services. + +```rust +let entries = op.list_with(path).deleted(true).await; +``` + +Please note that `deleted` here means "including deleted files" rather than "only deleted files." Therefore, `list_with(path).deleted(true)` will list both current files and deleted ones. + Review Comment: Then `with_deleted` or `include_deleted`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
