yanbinyang opened a new issue, #712: URL: https://github.com/apache/paimon-rust/issues/712
### Search before asking - [x] I searched the existing issues. [#616](https://github.com/apache/paimon-rust/issues/616) and [#618](https://github.com/apache/paimon-rust/pull/618) added `FileIOBuilder::with_fs_operator`, but deliberately restricted it to filesystem semantics. Scheme-based object-store URLs are rejected because they need scheme/authority/bucket-aware path resolution. ### Motivation Embedding engines often own tenant-scoped, renewable cloud credentials. Passing a static AK/SK/token snapshot through FileIO properties is not sufficient because a Paimon `Table` / `FileIO` can outlive the credential. The priority external-table use cases are: - AWS S3: default credential chain followed by per-tenant AssumeRole, including session name and external ID. - Aliyun OSS: machine OIDC role -> per-tenant AssumeRole, followed by automatic STS AK/SK/security-token refresh. - GCS: VM/default service account -> target service-account impersonation. - Azure Blob / ADLS: broker-issued SAS credentials with expiration and refresh. These flows are application-specific and should not require paimon-rust to implement every cloud credential broker. However, the current closed `Storage` enum and the filesystem-only operator hook do not let an embedding application provide a refreshable object-store implementation. This is currently blocking full delegated-credential support for Paimon external tables in milvus-storage. The related community request is documented in [milvus-storage#594](https://github.com/milvus-io/milvus-storage/pull/594#discussion_r3663598035). ### Reference: iceberg-rust iceberg-rust 0.9 exposes two public extension points: - async [`Storage`](https://github.com/apache/iceberg-rust/blob/7ef4063926f76f4ab3037227a9fa7a53e21e717f/crates/iceberg/src/io/storage/mod.rs) - [`StorageFactory`](https://github.com/apache/iceberg-rust/blob/7ef4063926f76f4ab3037227a9fa7a53e21e717f/crates/iceberg/src/io/storage/mod.rs#L103-L138), injected through [`FileIOBuilder::new(factory)`](https://github.com/apache/iceberg-rust/blob/7ef4063926f76f4ab3037227a9fa7a53e21e717f/crates/iceberg/src/io/file_io.rs#L171-L221) `FileIO` caches the constructed storage, while the custom storage can retain a renewable provider and resolve/rebuild its delegated inner storage during async I/O. milvus-storage uses this design while retaining OpenDAL for the actual object-store operations: - [Iceberg factory selection and injection](https://github.com/milvus-io/milvus-storage/blob/e759b53b6bb8435596cb1244865c35b8ab1ad8fa/cpp/src/format/bridge/rust/src/iceberg_bridgeimpl.rs) - [AWS AssumeRole credential loader](https://github.com/milvus-io/milvus-storage/blob/e759b53b6bb8435596cb1244865c35b8ab1ad8fa/cpp/src/format/bridge/rust/src/aws_arn_provider.rs) - [Aliyun two-step OIDC/AssumeRole storage](https://github.com/milvus-io/milvus-storage/blob/e759b53b6bb8435596cb1244865c35b8ab1ad8fa/cpp/src/format/bridge/rust/src/aliyun_oss_provider.rs) - [GCS impersonation storage](https://github.com/milvus-io/milvus-storage/blob/e759b53b6bb8435596cb1244865c35b8ab1ad8fa/cpp/src/format/bridge/rust/src/gcp_impersonation.rs) - [Azure broker SAS storage](https://github.com/milvus-io/milvus-storage/blob/e759b53b6bb8435596cb1244865c35b8ab1ad8fa/cpp/src/format/bridge/rust/src/azure_sas_provider.rs) OpenDAL is still the storage substrate. The factory is needed above OpenDAL because not every backend exposes the same dynamic credential-provider hook; for token-based flows, the custom storage may need to rebuild and atomically swap the OpenDAL operator when credentials rotate. ### Solution Add a public object-store extension point analogous to iceberg-rust, while preserving `with_fs_operator` for its existing filesystem use case. One possible API shape is: ```rust #[async_trait] pub trait FileIOStorage: Debug + Send + Sync { // async storage operations using the full absolute object-store URI // ... } pub trait FileIOStorageFactory: Debug + Send + Sync { fn build(&self, config: &StorageConfig) -> Result<Arc<dyn FileIOStorage>>; } impl FileIOBuilder { pub fn with_storage_factory( self, factory: Arc<dyn FileIOStorageFactory>, ) -> Self; } ``` A smaller alternative could be an async, scheme-aware `ObjectStoreOperatorProvider` returning a resolved `Operator` plus relative object path. The important properties are: 1. The extension receives the full URI, or a parsed location containing scheme, authority/bucket and object path. It must not reuse filesystem path resolution. 2. Provider/storage state is retained by `FileIO` and shared by table/scan readers. 3. Credential lookup and refresh can happen during async I/O, without rebuilding the Paimon `Table` or calling `get_table` again. 4. The implementation can reuse an operator while credentials are fresh and atomically replace it after refresh. 5. Explicit delegated authentication fails closed; it must not silently fall back to another ambient identity. 6. Credentials are excluded from `Debug`, serialization and cache keys. Cache/provider scope must include tenant identity and storage authority, not only the bucket name. 7. Built-in OpenDAL-backed storage remains the default and custom factories may delegate normal I/O to OpenDAL. The factory/provider should be generic. AWS STS, Aliyun OIDC, GCP IAM Credentials, and Azure credential-broker protocols should remain in embedding applications unless a reusable implementation belongs in OpenDAL. ### Suggested acceptance tests - A custom factory correctly resolves absolute `s3://`, `oss://`, `gs://`, and Azure Blob/ADLS URLs. - A fake expiring provider rotates credentials while the same `FileIO` remains alive. - Concurrent refresh is single-flight. - Cross-scheme and cross-authority/bucket reuse is rejected. - Existing built-in storage and `with_fs_operator` behavior remains unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
