mixermt opened a new issue, #6109: URL: https://github.com/apache/datafusion-comet/issues/6109
### What is the problem the feature request solves? With #6106 every task on an executor shares one `FileIO` per storage configuration. What that sharing buys depends on what the backend's `Storage` holds. At the pinned iceberg-rust rev (`665c64e`), `OpenDalStorage::S3`, `Gcs` and `Oss` hold only the parsed config and, for S3, the access loader. `create_operator` in `iceberg-storage-opendal` builds a new opendal `Operator` on every `new_input`, `new_output`, `exists` and `metadata`, and `opendal-service-s3`'s `build()` constructs a new `Signer` for each operator. The signer is where reqsign keeps the access cache, so with a custom S3 access provider every file open still pays a JNI `provide_credential` round trip, and every operator re-parses and re-validates its configuration. HTTP connection pooling is unaffected either way: opendal's default transport is process-wide. For the S3 family, then, #6106 reuses the factory, the config and the bridge, but not the client or signer. The HDFS backend in #5898 already solves this on its side: `OpenDalStorage::HdfsNative` carries an operator cache keyed by NameNode, so the hdfs-native client is built once per `Storage` and, with #6106, once per executor. ### Describe the potential solution Apply the HDFS backend's pattern to the S3 family in `iceberg-storage-opendal`: an operator cache inside `OpenDalStorage::S3` (and `Gcs`, `Oss`) keyed by the operator's root, so `create_operator` returns an existing `Operator` for a bucket it has already built. Since `Storage` lives as long as the cached `FileIO`, the signer and its access cache would then be shared by every task on the executor, and the per-open JNI fetch collapses to one per access expiry. This is an upstream change in `apache/iceberg-rust`, followed by a pin bump here. Comet cannot do it locally: `create_operator` is `pub(crate)` and the `Storage` trait exposes no operators. ### Additional context Split out of #6105 at review time so it stays open after #6106 merges. The operator cache is only effective once a `FileIO` outlives a task, which is what #6106 provides. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
