[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17280997#comment-17280997
]
Mehakmeet Singh commented on HADOOP-13887:
------------------------------------------
I have been trying to add support for CSE using the
AmazonS3EncryptionClientV2Builder in DefaultS3ClientFactory, but met with
Region issues like :
{code:java}
2021-02-08 15:37:01,675 [setup] WARN util.EC2MetadataUtils
(EC2MetadataUtils.java:getItems(410)) - Unable to retrieve the requested
metadata (/latest/dynamic/instance-identity/document). Failed to connect to
service endpoint: 2021-02-08 15:37:01,675 [setup] WARN util.EC2MetadataUtils
(EC2MetadataUtils.java:getItems(410)) - Unable to retrieve the requested
metadata (/latest/dynamic/instance-identity/document). Failed to connect to
service endpoint: com.amazonaws.SdkClientException: Failed to connect to
service endpoint: at
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
at
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
at
com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
at
com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403) at
com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:372) at
com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:368) at
com.amazonaws.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:283)
at
com.amazonaws.regions.InstanceMetadataRegionProvider.tryDetectRegion(InstanceMetadataRegionProvider.java:59)
at
com.amazonaws.regions.InstanceMetadataRegionProvider.getRegion(InstanceMetadataRegionProvider.java:50)
at
com.amazonaws.regions.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:46)
at
com.amazonaws.client.builder.AwsClientBuilder.determineRegionFromRegionProvider(AwsClientBuilder.java:475)
at
com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:458)
at
com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
at
com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at
com.amazonaws.services.s3.AmazonS3EncryptionClientV2.newAWSKMSClient(AmazonS3EncryptionClientV2.java:197)
at
com.amazonaws.services.s3.AmazonS3EncryptionClientV2.<init>(AmazonS3EncryptionClientV2.java:115)
at
com.amazonaws.services.s3.AmazonS3EncryptionClientV2Builder.build(AmazonS3EncryptionClientV2Builder.java:101)
at
com.amazonaws.services.s3.AmazonS3EncryptionClientV2Builder.build(AmazonS3EncryptionClientV2Builder.java:23)
at
com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.newAmazonS3EncryptionClient(DefaultS3ClientFactory.java:188)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:126)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:751) at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:444) at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3460) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:536) at
org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:187)
at
org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.lang.Thread.run(Thread.java:748)Caused by: java.net.ConnectException: No
route to host (connect failed) at java.net.PlainSocketImpl.socketConnect(Native
Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476) at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394) at
java.net.Socket.connect(Socket.java:606) at
sun.net.NetworkClient.doConnect(NetworkClient.java:175) at
sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at
sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at
sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at
sun.net.www.http.HttpClient.New(HttpClient.java:339) at
sun.net.www.http.HttpClient.New(HttpClient.java:357) at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at
com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)
at
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)
... 42 more2021-02-08 15:37:01,682 [setup] WARN fs.FileSystem
(FileSystem.java:createFileSystem(3464)) - Failed to initialize fileystem
s3a://mehakmeet-singh-data/: org.apache.hadoop.fs.s3a.AWSClientIOException:
initializing on s3a://mehakmeet-singh-data/: com.amazonaws.SdkClientException:
Unable to find a region via the region provider chain. Must provide an explicit
region in the builder or setup environment to supply a region.: Unable to find
a region via the region provider chain. Must provide an explicit region in the
builder or setup environment to supply a region.2021-02-08 15:37:01,684
[teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(278)) - closing file system
org.apache.hadoop.fs.s3a.AWSClientIOException: initializing on
s3a://mehakmeet-singh-data/: com.amazonaws.SdkClientException: Unable to find a
region via the region provider chain. Must provide an explicit region in the
builder or setup environment to supply a region.: Unable to find a region via
the region provider chain. Must provide an explicit region in the builder or
setup environment to supply a region.
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170) at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:518) at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3460) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:536) at
org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:187)
at
org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.lang.Thread.run(Thread.java:748)Caused by:
com.amazonaws.SdkClientException: Unable to find a region via the region
provider chain. Must provide an explicit region in the builder or setup
environment to supply a region. at
com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462)
{code}
So, basically getting Region issues while trying to implement
AmazonS3EncryptionClientV2 for CSE and trying to test it. I have also Debugged
this, and I was able to see that the endpoint: *s3.ap-south-1.amazonaws.com*
for my bucket was being propogated, but still the error persists.
> Encrypt S3A data client-side with AWS SDK (S3-CSE)
> --------------------------------------------------
>
> Key: HADOOP-13887
> URL: https://issues.apache.org/jira/browse/HADOOP-13887
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Jeeyoung Kim
> Assignee: Igor Mazur
> Priority: Minor
> Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch,
> HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch,
> HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch,
> HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch,
> HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch,
> HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch,
> HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf
>
>
> Expose the client-side encryption option documented in Amazon S3
> documentation -
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html
> Currently this is not exposed in Hadoop but it is exposed as an option in AWS
> Java SDK, which Hadoop currently includes. It should be trivial to propagate
> this as a parameter passed to the S3client used in S3AFileSystem.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
