[
https://issues.apache.org/jira/browse/HADOOP-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648798#comment-17648798
]
ASF GitHub Bot commented on HADOOP-18579:
-----------------------------------------
virajjasani commented on code in PR #5230:
URL: https://github.com/apache/hadoop/pull/5230#discussion_r1051156833
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/V2Migration.java:
##########
@@ -95,4 +99,13 @@ public static void v1GetObjectMetadataCalled() {
+ "will be changed as part of upgrading S3A to AWS SDK V2");
}
+ /**
+ * Warns on use of getObjectMetadata.
+ */
+ public static void regionNotConfigured() {
+ WARN_ON_REGION_NOT_CONFIGURED.warn("A region has not been configured.
Cross region support "
+ + "will be removed as part of upgrading S3A to AWS SDK V2. To
avoid errors, set the "
+ + "bucket's region in {}.", AWS_REGION);
+ }
Review Comment:
How about passing region name to the method and let the method decide
whether to warn?
```
public static void warnIfRegionNotConfigured(String regionName) {
if (StringUtils.isBlank(regionName)) {
WARN_ON_REGION_NOT_CONFIGURED.warn("A region has not been configured.
Cross region support "
+ "will be removed as part of upgrading S3A to AWS SDK V2. To
avoid errors, set the "
+ "bucket's region in {}.", AWS_REGION);
}
}
```
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##########
@@ -234,6 +236,12 @@ protected AmazonS3 buildAmazonS3Client(
AmazonS3ClientBuilder b = AmazonS3Client.builder();
configureBasicParams(b, awsConf, parameters);
+ String region = getConf().getTrimmed(AWS_REGION);
+
+ if (StringUtils.isBlank(region)) {
+ V2Migration.regionNotConfigured();
+ }
+
// endpoint set up is a PITA
AwsClientBuilder.EndpointConfiguration epr
= createEndpointConfiguration(parameters.getEndpoint(),
Review Comment:
nit: `getConf().getTrimmed(AWS_REGION)` passed to
`createEndpointConfiguration()` can be replaced with `region`
> Warn when no region is configured
> ---------------------------------
>
> Key: HADOOP-18579
> URL: https://issues.apache.org/jira/browse/HADOOP-18579
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.9
> Reporter: Ahmar Suhail
> Priority: Minor
> Labels: pull-request-available
>
> The AWS Java SDK V1 allows for cross region access. This means that even if
> you instantiate the S3 client with US_EAST_1 (or any region different to your
> actual bucket's region), the SDK will figure out the region.
>
> With the upgrade to SDK V2, this is no longer supported and the region should
> be set explicitly. Requests with the incorrect region will fail. To prepare
> for this change, S3A should warn when a region is not set via
> fs.s3a.endpoint.region.
>
> We should warn even if fs.s3a.endpoint is set and region can be parsed from
> this. This is because it is recommended to let the SDK V2 figure out the
> endpoint to use from the region, and so S3A should discourage from setting
> the endpoint unless absolutely required (eg for third party stores).
>
> Ideally rename fs.s3a.endpoint.region to fs.s3a.region, but not sure if this
> is ok to do.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
