universalmind303 opened a new issue, #7194: URL: https://github.com/apache/opendal/issues/7194
### Feature Description Enable third-party applications to register custom OpenDAL services (implementations of Access) across FFI boundaries, so that external Rust crates compiled as separate shared libraries can provide new storage backends without compile-time coupling. **Desired outcome** An extension author can: 1. Implement an OpenDAL service in Rust (following the standard Access pattern, like https://github.com/apache/opendal/tree/main/core/services/huggingface) 2. Compile it as a standalone shared library (no compile-time dependency on the host application) 3. Register it with the host application at runtime via some ABI-stable mechanism 4. The host application wraps it as a normal Operator with its own layers (retry, metrics, etc.) The existing C-ABI types (opendal_bytes, opendal_error, opendal_metadata, etc.) seem like natural building blocks for this. ### Problem and Solution We're integrating OpenDAL into https://github.com/Eventual-Inc/Daft (a distributed dataframe engine) to support pluggable storage backends. Our users want to implement custom protocols (e.g. `my-proprietary-protocol://`) by writing an OpenDAL service in Rust, packaging it as a Python wheel (cdylib via PyO3/maturin), and registering it at runtime. The challenge is that Daft and the extension are separate shared libraries loaded into the same Python process. They each statically link their own copy of OpenDAL, so Rust types can't be safely shared across the dylib boundary. Current state: OpenDAL's C bindings let you consume services (call opendal_operator_read, etc.), but there's no way to implement a service from the other side of an FFI boundary. The OperatorRegistry requires compile-time generics (register<B: Builder>), so you can't register a service from a separately compiled shared library. ### Additional Context Use cases - Daft — pluggable storage protocols for a dataframe engine _(the motivating use case)_ - Any application embedding OpenDAL that wants to allow third-party storage plugins without recompilation - Cross-language service implementations — implement an OpenDAL service in Go, Python (via cffi), etc. ### Are you willing to contribute to the development of this feature? - [ ] Yes, I am willing to contribute to the development of this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
