dwsmith1983 commented on code in PR #5872:
URL: https://github.com/apache/datafusion-comet/pull/5872#discussion_r4081277196
##########
native/core/src/parquet/objectstore/s3.rs:
##########
@@ -239,24 +240,29 @@ fn extract_s3_config_options(
s3_configs.insert(AmazonS3ConfigKey::Region, region.to_string());
}
- // Extract and handle path style access (virtual hosted style)
- let mut virtual_hosted_style_request = false;
- if let Some(path_style) = get_config_trimmed(configs, bucket,
"path.style.access") {
- virtual_hosted_style_request = path_style.to_lowercase() == "true";
- s3_configs.insert(
- AmazonS3ConfigKey::VirtualHostedStyleRequest,
- virtual_hosted_style_request.to_string(),
- );
- }
+ // Hadoop defaults fs.s3a.path.style.access to false, which means
virtual-hosted addressing,
+ // and treats non-boolean text as that default. object_store expects the
inverse flag.
+ let path_style_access = get_config_trimmed(configs, bucket,
"path.style.access")
+ .is_some_and(|value| value.eq_ignore_ascii_case("true"));
+ // The AWS SDK addresses a bucket whose name contains a dot path-style
over HTTPS, because
+ // the dotted host does not match S3's wildcard certificate. The default
AWS endpoint is
+ // HTTPS, and normalize_endpoint applies the same rule to a custom one by
its scheme.
+ let mut virtual_hosted_style_request =
+ !path_style_access && !bucket_needs_path_style_over_https(bucket);
Review Comment:
> Please preserve virtual hosting for this HTTP case and cover it through
`extract_s3_config_options`, including the resulting URL.
In 31fafaffd. The dotted-bucket rule runs in `extract_s3_config_options`
only when no custom endpoint is configured, since the default endpoint is
HTTPS; a custom endpoint decides by its own scheme inside `normalize_endpoint`.
The test goes through `extract_s3_config_options` and asserts
`http://review.dotted.bucket.storage.example.test` for the HTTP case beside the
HTTPS and default cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
