This is an automated email from the ASF dual-hosted git repository. xuanwo pushed a commit to branch xuanwo/services-split-plan in repository https://gitbox.apache.org/repos/asf/opendal.git
commit a3a915ac4709d02d6b18c0047b830f12a0f4a85e Author: Xuanwo <[email protected]> AuthorDate: Thu Nov 27 22:01:19 2025 +0800 rfc: core Signed-off-by: Xuanwo <[email protected]> --- core/src/docs/rfcs/0000_services_layers_split.md | 108 +++++++++++++++++++++++ core/src/docs/rfcs/mod.rs | 4 + 2 files changed, 112 insertions(+) diff --git a/core/src/docs/rfcs/0000_services_layers_split.md b/core/src/docs/rfcs/0000_services_layers_split.md new file mode 100644 index 000000000..00713bdae --- /dev/null +++ b/core/src/docs/rfcs/0000_services_layers_split.md @@ -0,0 +1,108 @@ +- Proposal Name: split_services_layers +- Start Date: 2025-11-27 +- RFC PR: [apache/opendal#7000](https://github.com/apache/opendal/pull/7000) +- Tracking Issue: [apache/opendal#7000](https://github.com/apache/opendal/issues/7000) + +# Summary + +Introduce a crate topology where `opendal-core` owns types, raw traits, and a minimal set of critical layers; each storage backend lives in its own `opendal-service-{backend}` crate; non-critical layers move to `opendal-layer-{name}` crates; and a facade crate `opendal` re-exports core, services, and layers while wiring dynamic service registration and feature aliases. + +# Motivation + +- Decouple compilation units so users only build the services and layers they need, reducing compile time and dependency footprint. +- Make service delivery and versioning independent, enabling faster backend evolution without touching core. +- Allow layer evolution at its own pace while keeping correctness-critical layers close to core for safety. +- Prepare for future plugin ecosystems (e.g., third-party services/layers published independently) without breaking existing API paths. + +# Guide-level explanation + +As a user you depend on the facade crate `opendal` as before. Differences: + +- To enable a backend, turn on the corresponding feature, for example `opendal = { version = "x.y", features = ["service-s3"] }`. The feature pulls in `opendal-service-s3`. +- Critical layers (error context, simulate, completeness, correctness check, capability check, immutable index, HTTP client) are always available via `opendal::layers` because they live in `opendal-core`. +- Non-critical layers are opt-in features, e.g., `"layer-logging"` pulls `opendal-layer-logging` and re-exports `LoggingLayer` under `opendal::layers::LoggingLayer`. +- API surface stays the same: `opendal::Operator`, `opendal::services::S3`, `opendal::layers::LoggingLayer` still work; only crate boundaries change under the hood. + +Migration for existing users: + +```toml +# Old +opendal = { version = "0.55", features = ["services-s3", "layers-logging"] } + +# New (facade keeps aliases for one major version) +opendal = { version = "0.56", features = ["service-s3", "layer-logging"] } +``` + +# Reference-level explanation + +## Crate layout + +- `opendal-core` + - Modules: `types`, `raw`, `blocking`, `docs`, critical layers (ErrorContext, Simulate, Complete, CorrectnessCheck, CapabilityCheck, ImmutableIndex, HttpClient, optional TailCut). + - No service implementations, no non-critical layers. + - Exposes `OperatorRegistry` and `Scheme` but performs no builtin registrations. +- `opendal-service-{backend}` + - Contains the backend’s `Builder`, `Config`, accessor implementation, scheme constant, and a `register(&OperatorRegistry)` helper. + - Depends only on `opendal-core` (plus backend-specific deps). +- `opendal-layer-{name}` + - Each non-critical layer in its own crate (or small groups when strongly coupled, e.g., oteltrace/otelmetrics). + - Depends on `opendal-core` and layer-specific deps. +- `opendal` facade + - Depends on `opendal-core` (mandatory), `opendal-layer-*` and `opendal-service-*` as optional deps behind features. + - Re-exports `core::{Operator, Result, ...}`, critical layers, and optional services/layers. + - On build, calls `register_builtin_services` that invokes each enabled service crate’s `register` function to populate `DEFAULT_OPERATOR_REGISTRY`. + - Maintains feature aliases for one major version (e.g., `services-s3` -> `service-s3`, `layers-logging` -> `layer-logging`). + +## Feature matrix + +- Core features: keep executor/runtime toggles (`executors-tokio`, `blocking`), internal helpers (`internal-path-cache`, `internal-tokio-rt`), plus minimal TLS/http knobs. +- Service features: `service-{backend}` live in facade, map 1:1 to `opendal-service-{backend}` optional dependency. +- Layer features: `layer-{name}` live in facade, map 1:1 to `opendal-layer-{name}` optional dependency; critical layers require no feature. + +## Registry and Scheme + +- `Operator::from_uri` remains, but core contains no builtin registrations. +- Facade constructs registry at init by conditionally calling each service crate’s `register` based on enabled features. +- `Scheme::enabled()` moves to facade to aggregate enabled services; core provides base enum and helper for custom schemes. + +## Testing and CI + +- Core tests remain for types/raw/critical layers. +- Each service crate carries unit tests plus behavior tests triggered via shared harness; a new `opendal-behavior` test crate can reuse the existing fixtures. +- Each layer crate keeps its own minimal tests; integration tests that combine layers/services run in facade or dedicated smoke suites. +- Workspace-level tasks (`cargo clippy`, `cargo fmt --check`) cover all crates; behavior matrix splits by service features to control runtime cost. + +## Compatibility + +- Public API paths stay stable through facade re-exports. +- Feature aliases maintained for one major release; deprecation warnings in docs and cargo feature descriptions. +- Crate names and versions remain under Apache OpenDAL namespace. + +# Drawbacks + +- More crates increase release coordination overhead and CI matrix size. +- Users pinning to core-only without facade must manually enable services/layers they need. +- Potential longer resolve time in dependency solvers due to many optional dependencies. + +# Rationale and alternatives + +- Keeping critical layers in core avoids breaking invariants and reduces dependency churn on safety-sensitive code. +- Per-layer and per-service crates maximize compile-time selectivity and allow independent evolution. +- Alternative considered: one `opendal-layers` mega-crate; rejected to keep feature granularity and lighter dependency pull for single-layer use. + +# Prior art + +- Rust ecosystem crates like `tokio` vs `tokio-util`, `tower` vs `tower-*` split utilities from core runtime. +- Storage SDKs (AWS SDK for Rust) split per-service crates to reduce bloat and compile time. + +# Unresolved questions + +- Exact set of critical layers that must stay in core (TailCut, Retry?) may need tuning after performance and safety review. +- Whether to group strongly coupled layers (otelmetrics/oteltrace) into one crate or keep fully separate. +- Behavior test harness packaging: shared crate vs duplicated fixtures. + +# Future possibilities + +- Allow third-party service or layer crates to self-register via cargo features or a plugin registry. +- Provide a `minimal` facade profile that ships only core + registry glue for embedders. +- Introduce codegen to auto-wire service registrations to shrink manual glue in facade. diff --git a/core/src/docs/rfcs/mod.rs b/core/src/docs/rfcs/mod.rs index 36cdaa6b2..2d72d0fe1 100644 --- a/core/src/docs/rfcs/mod.rs +++ b/core/src/docs/rfcs/mod.rs @@ -288,3 +288,7 @@ pub mod rfc_6707_capability_override_layer {} /// Checksum #[doc = include_str!("6817_checksum.md")] pub mod rfc_6817_checksum {} + +/// Services and layers split topology +#[doc = include_str!("7000_services_layers_split.md")] +pub mod rfc_7000_services_layers_split {}
