This is an automated email from the ASF dual-hosted git repository.
xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opendal.git
The following commit(s) were added to refs/heads/main by this push:
new b735beaa8 feat(services/s3): add configuration aliases for better
Arrow object_store compatibility (#6524)
b735beaa8 is described below
commit b735beaa8b44090eaeac19cc17335fc1c83ec2df
Author: Jack Ye <[email protected]>
AuthorDate: Mon Aug 25 10:29:51 2025 -0700
feat(services/s3): add configuration aliases for better Arrow object_store
compatibility (#6524)
* feat(services/s3): add configuration aliases for better AWS SDK
compatibility
Add serde aliases to S3Config fields to support multiple configuration key
naming conventions.
This enables seamless migration from AWS SDK and other S3-compatible
libraries by allowing
users to use either OpenDAL's native field names or AWS-prefixed
alternatives.
## Added Aliases
| OpenDAL Field | Added Aliases | Arrow Reference |
|---------------|---------------|-----------------|
| `bucket` | `aws_bucket`, `aws_bucket_name`, `bucket_name` |
[L1026](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1026)
|
| `access_key_id` | `aws_access_key_id` |
[L1022](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1022)
|
| `secret_access_key` | `aws_secret_access_key` |
[L1023](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1023)
|
| `session_token` | `aws_session_token`, `aws_token`, `token` |
[L1028](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1028)
|
| `region` | `aws_region` |
[L1025](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1025)
|
| `endpoint` | `aws_endpoint`, `aws_endpoint_url`, `endpoint_url` |
[L1027](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1027)
|
| `enable_virtual_host_style` | `aws_virtual_hosted_style_request`,
`virtual_hosted_style_request` |
[L1029-L1030](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1029-L1030)
|
| `checksum_algorithm` | `aws_checksum_algorithm` |
[L1036](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1036)
|
| `enable_request_payer` | `aws_request_payer`, `request_payer` |
[L1044](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1044)
|
| `server_side_encryption` | `aws_server_side_encryption` |
[L1047-L1048](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1047-L1048)
|
| `server_side_encryption_aws_kms_key_id` | `aws_sse_kms_key_id` |
[L1049](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1049)
|
| `server_side_encryption_customer_key` | `aws_sse_customer_key_base64` |
[L1053-L1054](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1053-L1054)
|
## Missing Configurations (Future Work)
These Arrow configurations are not yet available in OpenDAL:
| Arrow Configuration | Aliases | Reference | Description |
|-------------------|---------|-----------|-------------|
| `DefaultRegion` | `aws_default_region`, `default_region` |
[L1024](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1024)
| Fallback region when region not specified |
| `ImdsV1Fallback` | `aws_imdsv1_fallback`, `imdsv1_fallback` |
[L1033](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1033)
| Allow fallback to IMDSv1 for credentials |
| `UnsignedPayload` | `aws_unsigned_payload`, `unsigned_payload` |
[L1035](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1035)
| Skip payload signing for performance |
| `MetadataEndpoint` | `aws_metadata_endpoint`, `metadata_endpoint` |
[L1034](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1034)
| Custom metadata service endpoint |
| `ContainerCredentialsRelativeUri` |
`aws_container_credentials_relative_uri` |
[L1037](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1037)
| ECS container credentials |
| `ContainerCredentialsFullUri` | `aws_container_credentials_full_uri` |
[L1038](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1038)
| EKS pod credentials |
| `ContainerAuthorizationTokenFile` |
`aws_container_authorization_token_file` |
[L1039](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1039)
| EKS auth token file |
| `CopyIfNotExists` | `aws_copy_if_not_exists`, `copy_if_not_exists` |
[L1041](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1041)
| Conditional copy operations |
| `ConditionalPut` | `aws_conditional_put`, `conditional_put` |
[L1042](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1042)
| Conditional put operations |
| `SkipSignature` | `aws_skip_signature`, `skip_signature` |
[L1040](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1040)
| Skip request signing |
| `DisableTagging` | `aws_disable_tagging`, `disable_tagging` |
[L1043](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1043)
| Disable object tagging |
| `S3Express` | `aws_s3_express`, `s3_express` |
[L1032](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1032)
| S3 Express One Zone support |
| `BucketKeyEnabled` | `aws_sse_bucket_key_enabled` |
[L1051-L1052](https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L1051-L1052)
| SSE-KMS bucket key optimization |
Contributes to #6456
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* fix: resolve clippy and formatting issues
- Fix clippy::bool_assert_comparison warnings by using assert!() instead of
assert_eq!() with boolean literals
- Apply rustfmt formatting for multi-line serde attributes
- All tests continue to pass
* test(s3): split test_s3_config_aliases into individual tests per config
- Split the combined test into 4 separate tests for better clarity
- test_s3_config_original_field_names: Tests original field names
- test_s3_config_aws_prefixed_aliases: Tests AWS-prefixed aliases
- test_s3_config_additional_aliases: Tests additional alias variations
- test_s3_config_encryption_aliases: Tests encryption-related aliases
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
---------
Co-authored-by: Claude <[email protected]>
---
core/src/services/s3/config.rs | 115 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)
diff --git a/core/src/services/s3/config.rs b/core/src/services/s3/config.rs
index 1de80952d..0b371db2d 100644
--- a/core/src/services/s3/config.rs
+++ b/core/src/services/s3/config.rs
@@ -35,6 +35,7 @@ pub struct S3Config {
/// bucket name of this backend.
///
/// required.
+ #[serde(alias = "aws_bucket", alias = "aws_bucket_name", alias =
"bucket_name")]
pub bucket: String,
/// is bucket versioning enabled for this bucket
pub enable_versioning: bool,
@@ -54,6 +55,11 @@ pub struct S3Config {
/// - If endpoint is set, we will take user's input first.
/// - If not, we will try to load it from environment.
/// - If still not set, default to `https://s3.amazonaws.com`.
+ #[serde(
+ alias = "aws_endpoint",
+ alias = "aws_endpoint_url",
+ alias = "endpoint_url"
+ )]
pub endpoint: Option<String>,
/// Region represent the signing region of this endpoint. This is required
/// if you are using the default AWS S3 endpoint.
@@ -61,22 +67,26 @@ pub struct S3Config {
/// If using a custom endpoint,
/// - If region is set, we will take user's input first.
/// - If not, we will try to load it from environment.
+ #[serde(alias = "aws_region")]
pub region: Option<String>,
/// access_key_id of this backend.
///
/// - If access_key_id is set, we will take user's input first.
/// - If not, we will try to load it from environment.
+ #[serde(alias = "aws_access_key_id")]
pub access_key_id: Option<String>,
/// secret_access_key of this backend.
///
/// - If secret_access_key is set, we will take user's input first.
/// - If not, we will try to load it from environment.
+ #[serde(alias = "aws_secret_access_key")]
pub secret_access_key: Option<String>,
/// session_token (aka, security token) of this backend.
///
/// This token will expire after sometime, it's recommended to set
session_token
/// by hand.
+ #[serde(alias = "aws_session_token", alias = "aws_token", alias = "token")]
pub session_token: Option<String>,
/// role_arn for this backend.
///
@@ -106,6 +116,7 @@ pub struct S3Config {
/// server_side_encryption for this backend.
///
/// Available values: `AES256`, `aws:kms`.
+ #[serde(alias = "aws_server_side_encryption")]
pub server_side_encryption: Option<String>,
/// server_side_encryption_aws_kms_key_id for this backend
///
@@ -117,6 +128,7 @@ pub struct S3Config {
/// returned.
/// - If `server_side_encryption` is not `aws:kms`, setting
`server_side_encryption_aws_kms_key_id`
/// is a noop.
+ #[serde(alias = "aws_sse_kms_key_id")]
pub server_side_encryption_aws_kms_key_id: Option<String>,
/// server_side_encryption_customer_algorithm for this backend.
///
@@ -126,6 +138,7 @@ pub struct S3Config {
///
/// Value: BASE64-encoded key that matches algorithm specified in
/// `server_side_encryption_customer_algorithm`.
+ #[serde(alias = "aws_sse_customer_key_base64")]
pub server_side_encryption_customer_key: Option<String>,
/// Set server_side_encryption_customer_key_md5 for this backend.
///
@@ -152,6 +165,10 @@ pub struct S3Config {
///
/// - By default, opendal will send API to
`https://s3.us-east-1.amazonaws.com/bucket_name`
/// - Enabled, opendal will send API to
`https://bucket_name.s3.us-east-1.amazonaws.com`
+ #[serde(
+ alias = "aws_virtual_hosted_style_request",
+ alias = "virtual_hosted_style_request"
+ )]
pub enable_virtual_host_style: bool,
/// Set maximum batch operations of this backend.
///
@@ -180,6 +197,7 @@ pub struct S3Config {
///
/// Available options:
/// - "crc32c"
+ #[serde(alias = "aws_checksum_algorithm")]
pub checksum_algorithm: Option<String>,
/// Disable write with if match so that opendal will not send write
request with if match headers.
///
@@ -195,6 +213,7 @@ pub struct S3Config {
pub disable_list_objects_v2: bool,
/// Indicates whether the client agrees to pay for the requests made to
the S3 bucket.
+ #[serde(alias = "aws_request_payer", alias = "request_payer")]
pub enable_request_payer: bool,
}
@@ -210,3 +229,99 @@ impl Debug for S3Config {
d.finish_non_exhaustive()
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_s3_config_original_field_names() {
+ let json = r#"{
+ "bucket": "test-bucket",
+ "access_key_id": "test-key",
+ "secret_access_key": "test-secret",
+ "region": "us-west-2",
+ "endpoint": "https://s3.amazonaws.com";,
+ "session_token": "test-token"
+ }"#;
+
+ let config: S3Config = serde_json::from_str(json).unwrap();
+ assert_eq!(config.bucket, "test-bucket");
+ assert_eq!(config.access_key_id, Some("test-key".to_string()));
+ assert_eq!(config.secret_access_key, Some("test-secret".to_string()));
+ assert_eq!(config.region, Some("us-west-2".to_string()));
+ assert_eq!(
+ config.endpoint,
+ Some("https://s3.amazonaws.com".to_string())
+ );
+ assert_eq!(config.session_token, Some("test-token".to_string()));
+ }
+
+ #[test]
+ fn test_s3_config_aws_prefixed_aliases() {
+ let json = r#"{
+ "aws_bucket": "test-bucket",
+ "aws_access_key_id": "test-key",
+ "aws_secret_access_key": "test-secret",
+ "aws_region": "us-west-2",
+ "aws_endpoint": "https://s3.amazonaws.com";,
+ "aws_session_token": "test-token"
+ }"#;
+
+ let config: S3Config = serde_json::from_str(json).unwrap();
+ assert_eq!(config.bucket, "test-bucket");
+ assert_eq!(config.access_key_id, Some("test-key".to_string()));
+ assert_eq!(config.secret_access_key, Some("test-secret".to_string()));
+ assert_eq!(config.region, Some("us-west-2".to_string()));
+ assert_eq!(
+ config.endpoint,
+ Some("https://s3.amazonaws.com".to_string())
+ );
+ assert_eq!(config.session_token, Some("test-token".to_string()));
+ }
+
+ #[test]
+ fn test_s3_config_additional_aliases() {
+ let json = r#"{
+ "bucket_name": "test-bucket",
+ "token": "test-token",
+ "endpoint_url": "https://s3.amazonaws.com";,
+ "virtual_hosted_style_request": true,
+ "aws_checksum_algorithm": "crc32c",
+ "request_payer": true
+ }"#;
+
+ let config: S3Config = serde_json::from_str(json).unwrap();
+ assert_eq!(config.bucket, "test-bucket");
+ assert_eq!(config.session_token, Some("test-token".to_string()));
+ assert_eq!(
+ config.endpoint,
+ Some("https://s3.amazonaws.com".to_string())
+ );
+ assert!(config.enable_virtual_host_style);
+ assert_eq!(config.checksum_algorithm, Some("crc32c".to_string()));
+ assert!(config.enable_request_payer);
+ }
+
+ #[test]
+ fn test_s3_config_encryption_aliases() {
+ let json = r#"{
+ "bucket": "test-bucket",
+ "aws_server_side_encryption": "aws:kms",
+ "aws_sse_kms_key_id": "test-kms-key",
+ "aws_sse_customer_key_base64": "dGVzdC1jdXN0b21lci1rZXk="
+ }"#;
+
+ let config: S3Config = serde_json::from_str(json).unwrap();
+ assert_eq!(config.bucket, "test-bucket");
+ assert_eq!(config.server_side_encryption, Some("aws:kms".to_string()));
+ assert_eq!(
+ config.server_side_encryption_aws_kms_key_id,
+ Some("test-kms-key".to_string())
+ );
+ assert_eq!(
+ config.server_side_encryption_customer_key,
+ Some("dGVzdC1jdXN0b21lci1rZXk=".to_string())
+ );
+ }
+}
