wenym1 opened a new issue, #4286: URL: https://github.com/apache/opendal/issues/4286
## Motivation We are using AWS S3 or any S3 compatible object store as our storage backend. Previously we use [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust/tree/main/sdk/s3) to access the object store, and recently we are migrating from the aws sdk to opendal. Previously when we use the aws sdk, it supports passing customized http client (see method [http_client](https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/config/struct.Builder.html#method.http_client)). In our [code](https://github.com/risingwavelabs/risingwave/blob/15c403cdf5612306750c61bb53c030e8bb2207b7/src/object_store/src/object/s3.rs#L538), we build our own http client to apply some customized logic. Most logics can be supported in opendal, such as setting nodelay, keepalive. The only thing left is passing a customized connector. When building a [hyper client](https://durch.github.io/rust-goauth/hyper/client/struct.Client.html) by the [build](https://durch.github.io/rust-goauth/hyper/client/struct.Builder.html#method.build) method, it support passing a customized connector that implements the [Connect](https://durch.github.io/rust-goauth/hyper/client/connect/trait.Connect.html) trait. The trait is accessed when creating a new connection. We leverage this feature to monitor the t cp connection, such as number of existing connection, throughput of each connection etc. We implemented our own connector that wraps the original hyper http connector and impl the `Connect` trait in https://github.com/risingwavelabs/risingwave/pull/11848. However, in opendal this feature is not supported yet. ## Requested feature I think there are two ways to support the feature. The first one is to support passing a customized http client or a connector like what the aws sdk does. This may leak too much flexibility to our external users, as it depends on users to properly initialize the connector. The second one is to support passing a connector adapter with the following definition ``` trait ConnectorAdapter { fn wrap(connector: impl Connect) -> impl Connect; } ``` With this adapter trait, the opendal can initial the connector internally, and then wrap the initialized connector with this adapter, and in the end pass it to the http client. ## Implementation-wise consideration It seems that opendal is using reqwest to make the http request. I took a rough look at the code of reqwest. Though its underlying http client is also hyper, in [here](https://github.com/seanmonstar/reqwest/blob/68547a0af66222960eea6d649bf94497c11df246/src/async_impl/client.rs#L247), it seems that the connector used by the http client is fixed. So to support this feature, we shall either modify the reqwest code to support passing a customized connector, or change to make the http request from the raw hyper client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
