[
https://issues.apache.org/jira/browse/HADOOP-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788407#comment-17788407
]
Steve Loughran commented on HADOOP-18984:
-----------------------------------------
fyi, cloudstore now has a regions2 command to work this out. The classes we are
using are tagged as {{@SdkProtectedApi}} so we shouldn't be using them. but....
maybe we should actually support a chain of region determiners, which go from
bucket + endpoint => region
{code}
Determining region using SystemSettingsRegionProvider
=====================================================
System property aws.region
2023-11-21 12:45:44,339 [main] INFO sdk2.Regions2
(StoreDurationInfo.java:<init>(91)) - Starting:
SystemSettingsRegionProvider.getRegion()
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
2023-11-21 12:45:44,349 [main] INFO sdk2.Regions2
(StoreDurationInfo.java:close(193)) - Duration of
SystemSettingsRegionProvider.getRegion(): 0:00:00.011
WARNING: Provider SystemSettingsRegionProvider raised an exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region
from system settings. Region must be specified either via environment variable
(AWS_REGION) or system property (aws.region).
2023-11-21 12:45:44,350 [main] INFO sdk2.Regions2
(Regions2.java:printRegion(101)) - Provider raised an exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region
from system settings. Region must be specified either via environment variable
(AWS_REGION) or system property (aws.region).
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider.exception(SystemSettingsRegionProvider.java:41)
at java.util.Optional.orElseThrow(Optional.java:290)
at
software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider.getRegion(SystemSettingsRegionProvider.java:33)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.printRegion(Regions2.java:98)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.run(Regions2.java:67)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.exec(Regions2.java:125)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.main(Regions2.java:134)
at regions2.main(regions2.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
region is not known
Determining region using AwsProfileRegionProvider
=================================================
Region info in ~/.aws/config
2023-11-21 12:45:44,353 [main] INFO sdk2.Regions2
(StoreDurationInfo.java:<init>(91)) - Starting:
AwsProfileRegionProvider.getRegion()
2023-11-21 12:45:44,380 [main] INFO sdk2.Regions2
(StoreDurationInfo.java:close(193)) - Duration of
AwsProfileRegionProvider.getRegion(): 0:00:00.027
Region is determined as "eu-west-2"
Determining region using InstanceMetadataRegionProvider
=======================================================
EC2 metadata; will only work in AWS infrastructure
2023-11-21 12:45:44,380 [main] INFO sdk2.Regions2
(StoreDurationInfo.java:<init>(91)) - Starting:
InstanceMetadataRegionProvider.getRegion()
2023-11-21 12:45:45,741 [main] INFO sdk2.Regions2
(StoreDurationInfo.java:close(193)) - Duration of
InstanceMetadataRegionProvider.getRegion(): 0:00:01.360
WARNING: Provider InstanceMetadataRegionProvider raised an exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to contact EC2
metadata service.
2023-11-21 12:45:45,742 [main] INFO sdk2.Regions2
(Regions2.java:printRegion(101)) - Provider raised an exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to contact EC2
metadata service.
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:414)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:414)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:355)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:351)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:235)
at
software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider.tryDetectRegion(InstanceProfileRegionProvider.java:68)
at
software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider.getRegion(InstanceProfileRegionProvider.java:52)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.printRegion(Regions2.java:98)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.run(Regions2.java:73)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.exec(Regions2.java:125)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.main(Regions2.java:134)
at regions2.main(regions2.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:743)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:702)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1600)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1505)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:116)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.doReadResource(EC2MetadataUtils.java:419)
at
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:390)
... 20 more
region is not known
Region found: "eu-west-2"
=========================
Region was determined by AwsProfileRegionProvider as "eu-west-2"
{code}
> S3A: Support dynamic region resolution
> --------------------------------------
>
> Key: HADOOP-18984
> URL: https://issues.apache.org/jira/browse/HADOOP-18984
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> AWS v1 SDK used to do dynamic region inference including EC2 metadata. See
> HADOOP-17771 for that at work with the chain of: sysprops, env vars, EC2
> metadata.
> I think we *try* to do this, but I'm not sure it is working, maybe related to
> the region=null vs region="" options: its very hard to differentiate the two
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
