This is an automated email from the ASF dual-hosted git repository.
xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opendal.git
The following commit(s) were added to refs/heads/main by this push:
new 514abc0846 docs: Polish opendal rust core README (#4745)
514abc0846 is described below
commit 514abc08467fa762ca25cfa855088439ffceeae1
Author: Xuanwo <[email protected]>
AuthorDate: Mon Jun 17 14:32:44 2024 +0800
docs: Polish opendal rust core README (#4745)
Signed-off-by: Xuanwo <[email protected]>
---
README.md | 2 -
core/README.md | 243 ++++++++++++++++++++++++++++++++++-----------------------
2 files changed, 145 insertions(+), 100 deletions(-)
diff --git a/README.md b/README.md
index 5a8260a139..7e45c69cf6 100644
--- a/README.md
+++ b/README.md
@@ -115,8 +115,6 @@ OpenDAL offers a unified data access layer, empowering
users to seamlessly and e
| Cache Storage Services | [ghac] [memcached] [mini_moka] [moka]
[vercel_artifacts]
|
| Git Based Storage Services | [huggingface]
|
-> Welcome to add any services that are not currently supported
[here](https://github.com/apache/opendal/issues/5).
-
[sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02
[webdav]: https://datatracker.ietf.org/doc/html/rfc4918
diff --git a/core/README.md b/core/README.md
index 2b72a3fd96..6b577d6064 100644
--- a/core/README.md
+++ b/core/README.md
@@ -1,4 +1,4 @@
-# Apache OpenDAL™
+# Apache OpenDAL™: *Access Data Freely*
[![Build Status]][actions] [![Latest Version]][crates.io] [![Crate
Downloads]][crates.io] [![chat]][discord]
@@ -10,108 +10,149 @@
[chat]: https://img.shields.io/discord/1081052318650339399
[discord]: https://opendal.apache.org/discord
-**OpenDAL** is a data access layer that allows users to easily and efficiently
retrieve data from various storage services in a unified way.
+OpenDAL offers a unified data access layer, empowering users to seamlessly and
efficiently retrieve data from diverse storage services.
-- Documentation: [stable](https://docs.rs/opendal/) |
[main](https://opendal.apache.org/docs/rust/opendal/)
-- [Release
notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html)
-
-
-
-## Services
-
-<details>
-<summary>Standard Storage Protocols (like ftp, webdav)</summary>
-
-- ftp: FTP and FTPS
-- http: HTTP read-only services
-- sftp:
[SFTP](https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02)
services *being worked on*
-- webdav: [WebDAV](https://datatracker.ietf.org/doc/html/rfc4918) Service
-
-</details>
-
-<details>
-<summary>Object Storage Services (like s3, gcs, azblob)</summary>
-
-- azblob: [Azure Storage
Blob](https://azure.microsoft.com/en-us/services/storage/blobs/) services
-- cos: [Tencent Cloud Object
Storage](https://www.tencentcloud.com/products/cos) services
-- gcs: [Google Cloud Storage](https://cloud.google.com/storage) Service
-- obs: [Huawei Cloud Object
Storage](https://www.huaweicloud.com/intl/en-us/product/obs.html) Service (OBS)
-- oss: [Aliyun Object Storage Service](https://www.aliyun.com/product/oss)
(OSS)
-- s3: [AWS S3](https://aws.amazon.com/s3/) alike services
-- supabase: [Supabase Storage](https://supabase.com/docs/guides/storage)
Service *being worked on*
-
-</details>
-
-<details>
-<summary>File Storage Services (like fs, azdls, hdfs)</summary>
-
-- fs: POSIX alike file system
-- azdls: [Azure Data Lake Storage
Gen2](https://azure.microsoft.com/en-us/products/storage/data-lake-storage/)
services (As known as
[abfs](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver))
-- hdfs: [Hadoop Distributed File
System](https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)(HDFS)
-- ipfs: [InterPlanetary File System](https://ipfs.tech/) HTTP Gateway
-- ipmfs: [InterPlanetary File System](https://ipfs.tech/) MFS API *being
worked on*
-- webhdfs:
[WebHDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html)
Service
-
-</details>
-
-<details>
-<summary>Consumer Cloud Storage Service (like gdrive, onedrive, icloud
drive)</summary>
-
-- gdrive: [Google Drive](https://www.google.com/drive/) *being worked on*
-- onedrive:
[OneDrive](https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage)
*being worked on*
-- icloud: [Icloud Drive](https://www.icloud.com.cn/iclouddrive/) *being worked
on*
-
-</details>
+<img src="https://opendal.apache.org/img/architectural.png"; alt="OpenDAL
Architectural" width="61.8%" />
-<details>
-<summary>Key-Value Storage Service (like rocksdb, sled)</summary>
+## Useful Links
-- cacache: [cacache](https://crates.io/crates/cacache) backend
-- dashmap: [dashmap](https://github.com/xacrimon/dashmap) backend
-- memory: In memory backend
-- persy: [persy](https://crates.io/crates/persy) backend
-- redis: [Redis](https://redis.io/) services
-- rocksdb: [RocksDB](http://rocksdb.org/) services
-- sled: [sled](https://crates.io/crates/sled) backend
-- redb: [redb](https://crates.io/crates/redb) backend
-- tikv: [tikv](https://tikv.org/) backend
-- atomicserver:
[Atomicserver](https://github.com/atomicdata-dev/atomic-server) services
-
-</details>
-
-<details>
-<summary>Cache Storage Service (like memcached, moka)</summary>
-
-- ghac: [GitHub Action
Cache](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows)
Service
-- memcached: [Memcached](https://memcached.org/) service
-- mini_moka: [Mini Moka](https://github.com/moka-rs/mini-moka) backend
-- moka: [Moka](https://github.com/moka-rs/moka) backend
-- vercel_artifacts: [Vercel Remote
Caching](https://vercel.com/docs/concepts/monorepos/remote-caching) Service
*being worked on*
-
-</details>
-
-> Welcome to add any services that are not currently supported
[here](https://github.com/apache/opendal/issues/5).
-
-## Features
-
-Access data **freely**
-
-- Access different storage services in the same way
-- Behavior tests for all services
-
-Access data **painlessly**
-
-- **100%** documents covered
-- Powerful [`Layers`](https://docs.rs/opendal/latest/opendal/layers/index.html)
-- Automatic
[retry](https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html)
support
-- Full observability:
[logging](https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html),
[tracing](https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html),
[metrics](https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html).
-- [Native chaos
testing](https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html)
+- Documentation: [stable](https://docs.rs/opendal/) |
[main](https://opendal.apache.org/docs/rust/opendal/)
+- [Release
Notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html)
+- [Upgrade
Guide](https://docs.rs/opendal/latest/opendal/docs/upgrade/index.html)
+- [RFC List](https://docs.rs/opendal/latest/opendal/docs/rfcs/index.html)
-Access data **efficiently**
+## Services
-- Zero cost: Maps to API calls directly
-- Best effort: Automatically selects best read/seek/next based on services
-- Avoid extra calls: Reuses metadata when possible
+OpenDAL supports the following storage
[services](https://docs.rs/opendal/latest/opendal/services/index.html):
+
+| Type | Services
|
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
+| Standard Storage Protocols | ftp http [sftp] [webdav]
|
+| Object Storage Services | [azblob] [cos] [gcs] [obs] [oss] [s3] <br>
[b2] [openstack_swift] [upyun] [vercel_blob]
|
+| File Storage Services | fs [alluxio] [azdls] [azfile] [chainsafe]
[compfs] <br> [dbfs] [gridfs] [hdfs] [hdfs_native] [ipfs] [webhdfs]
|
+| Consumer Cloud Storage Service | [aliyun_drive] [gdrive] [onedrive]
[dropbox] [icloud] [koofr] <br> [pcloud] [seafile] [yandex_disk]
|
+| Key-Value Storage Services | [cacache] [cloudflare_kv] [dashmap] memory
[etcd] <br> [foundationdb] [persy] [redis] [rocksdb] [sled] <br> [redb] [tikv]
[atomicserver] |
+| Database Storage Services | [d1] [mongodb] [mysql] [postgresql]
[sqlite] [surrealdb]
|
+| Cache Storage Services | [ghac] [memcached] [mini_moka] [moka]
[vercel_artifacts]
|
+| Git Based Storage Services | [huggingface]
|
+
+[sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02
+[webdav]: https://datatracker.ietf.org/doc/html/rfc4918
+
+[azblob]: https://azure.microsoft.com/en-us/services/storage/blobs/
+[cos]: https://www.tencentcloud.com/products/cos
+[gcs]: https://cloud.google.com/storage
+[obs]: https://www.huaweicloud.com/intl/en-us/product/obs.html
+[oss]: https://www.aliyun.com/product/oss
+[s3]: https://aws.amazon.com/s3/
+[b2]: https://www.backblaze.com/
+[openstack_swift]: https://docs.openstack.org/swift/latest/
+[upyun]: https://www.upyun.com/
+[vercel_blob]: https://vercel.com/docs/storage/vercel-blob
+
+[alluxio]: https://docs.alluxio.io/os/user/stable/en/api/REST-API.html
+[azdls]: https://azure.microsoft.com/en-us/products/storage/data-lake-storage/
+[azfile]:
https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api
+[chainsafe]: https://storage.chainsafe.io/
+[compfs]: https://github.com/compio-rs/compio/
+[dbfs]: https://docs.databricks.com/en/dbfs/index.html
+[gridfs]: https://www.mongodb.com/docs/manual/core/gridfs/
+[hdfs]:
https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
+[hdfs_native]: https://github.com/Kimahriman/hdfs-native
+[ipfs]: https://ipfs.tech/
+[webhdfs]:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html
+
+[aliyun_drive]: https://www.aliyundrive.com/
+[gdrive]: https://www.google.com/drive/
+[onedrive]:
https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage
+[dropbox]: https://www.dropbox.com/
+[icloud]: https://www.icloud.com/iclouddrive
+[koofr]: https://koofr.eu/
+[pcloud]: https://www.pcloud.com/
+[seafile]: https://www.seafile.com/
+[yandex_disk]: https://360.yandex.com/disk/
+
+[cacache]: https://crates.io/crates/cacache
+[cloudflare_kv]: https://developers.cloudflare.com/kv/
+[dashmap]: https://github.com/xacrimon/dashmap
+[etcd]: https://etcd.io/
+[foundationdb]: https://www.foundationdb.org/
+[persy]: https://crates.io/crates/persy
+[redis]: https://redis.io/
+[rocksdb]: http://rocksdb.org/
+[sled]: https://crates.io/crates/sled
+[redb]: https://crates.io/crates/redb
+[tikv]: https://tikv.org/
+[atomicserver]: https://github.com/atomicdata-dev/atomic-server
+
+[d1]: https://developers.cloudflare.com/d1/
+[mongodb]: https://www.mongodb.com/
+[mysql]: https://www.mysql.com/
+[postgresql]: https://www.postgresql.org/
+[sqlite]: https://www.sqlite.org/
+[surrealdb]: https://surrealdb.com/
+
+[ghac]:
https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
+[memcached]: https://memcached.org/
+[mini_moka]: https://github.com/moka-rs/mini-moka
+[moka]: https://github.com/moka-rs/moka
+[vercel_artifacts]: https://vercel.com/docs/concepts/monorepos/remote-caching
+
+[huggingface]: https://huggingface.co/
+
+## Layers
+
+OpenDAL supports the following storage
[layers](https://docs.rs/opendal/latest/opendal/layers/index.html) to extend
the behavior:
+
+| Name | Depends | Description
|
+|---------------------------|------------------------|---------------------------------------------------------------------------------------|
+| [`AsyncBacktraceLayer`] | [async-backtrace] | Add Efficient, logical
'stack' traces of async functions for the underlying services. |
+| [`AwaitTreeLayer`] | [await-tree] | Add a Instrument
await-tree for actor-based applications to the underlying services. |
+| [`BlockingLayer`] | [tokio] | Add blocking API
support for non-blocking services. |
+| [`ChaosLayer`] | [rand] | Inject chaos into
underlying services for robustness test. |
+| [`ConcurrentLimitLayer`] | [tokio] | Add concurrent request
limit. |
+| [`DtraceLayer`] | [probe] | Support User
Statically-Defined Tracing(aka USDT) on Linux |
+| [`LoggingLayer`] | [log] | Add log for every
operations. |
+| [`MetricsLayer`] | [metrics] | Add metrics for every
operations. |
+| [`MinitraceLayer`] | [minitrace] | Add minitrace for every
operations. |
+| [`OtelTraceLayer`] | [opentelemetry::trace] | Add
opentelemetry::trace for every operations.
|
+| [`PrometheusClientLayer`] | [prometheus_client] | Add prometheus metrics
for every operations. |
+| [`PrometheusLayer`] | [prometheus] | Add prometheus metrics
for every operations. |
+| [`RetryLayer`] | [backon] | Add retry for temporary
failed operations. |
+| [`ThrottleLayer`] | [governor] | Add a bandwidth rate
limiter to the underlying services. |
+| [`TimeoutLayer`] | [tokio] | Add timeout for every
operations to avoid slow or unexpected hang operations. |
+| [`TracingLayer`] | [tracing] | Add tracing for every
operations. |
+
+[`AsyncBacktraceLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.AsyncBacktraceLayer.html
+[async-backtrace]: https://github.com/tokio-rs/async-backtrace
+[`AwaitTreeLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.AwaitTreeLayer.html
+[await-tree]: https://github.com/risingwavelabs/await-tree
+[`BlockingLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.BlockingLayer.html
+[tokio]: https://github.com/tokio-rs/tokio
+[`ChaosLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html
+[rand]: https://github.com/rust-random/rand
+[`ConcurrentLimitLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.ConcurrentLimitLayer.html
+[`DtraceLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.DtraceLayer.html
+[probe]: https://github.com/cuviper/probe-rs
+[`LoggingLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html
+[log]: https://github.com/rust-lang/log
+[`MetricsLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html
+[metrics]: https://github.com/metrics-rs/metrics
+[`MinitraceLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.MinitraceLayer.html
+[minitrace]: https://github.com/tikv/minitrace-rust
+[`OtelTraceLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.OtelTraceLayer.html
+[opentelemetry::trace]:
https://docs.rs/opentelemetry/latest/opentelemetry/trace/index.html
+[`PrometheusClientLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusClientLayer.html
+[prometheus_client]: https://github.com/prometheus/client_rust
+[`PrometheusLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusLayer.html
+[prometheus]: https://github.com/tikv/rust-prometheus
+[`RetryLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html
+[backon]: https://github.com/Xuanwo/backon
+[`ThrottleLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.ThrottleLayer.html
+[governor]: https://github.com/boinkor-net/governor
+[`TimeoutLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.TimeoutLayer.html
+[`TracingLayer`]:
https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html
+[tracing]: https://github.com/tokio-rs/tracing
## Quickstart
@@ -159,6 +200,12 @@ The examples are available at [here](../examples/rust).
Check out the [CONTRIBUTING](CONTRIBUTING.md) guide for more details on
getting started with contributing to this project.
+## Branding
+
+The first and most prominent mentions must use the full form: **Apache
OpenDAL™** of the name for any individual usage (webpage, handout, slides,
etc.) Depending on the context and writing style, you should use the full form
of the name sufficiently often to ensure that readers clearly understand the
association of both the OpenDAL project and the OpenDAL software product to the
ASF as the parent organization.
+
+For more details, see the [Apache Product Name Usage
Guide](https://www.apache.org/foundation/marks/guide).
+
## License and Trademarks
Licensed under the Apache License, Version 2.0:
http://www.apache.org/licenses/LICENSE-2.0
![https://opendal.apache.org/img/architectural.png"](https://opendal.apache.org/img/architectural.png"){kind=link}