[
https://issues.apache.org/jira/browse/HADOOP-13727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13727:
-----------------------------------
Attachment: HADOOP-13727-branch-2.005.patch
I'm attaching patch revision 005, rebasing to current.
bq. it passing a full test run against your endpoint
The rebased patch applied to branch-2 passes a full test run against us-west-2.
bq. you doing a look at the generated aws site page (or a github/markdown
editor) to make sure the 1. 2 ... bullet points are pulled out. I think you may
need an extra line.
Everything looks to be rendering nicely. These are actually part of Markdown
code blocks that repeat snippets of core-site.xml. Thanks for being watchful
of the documentation though.
bq. Ideally it'd be good to do the whole test suite to verify that IAM works
across the lot...
For this last test run, I relied on instance profile credentials for all of the
S3A tests. FWIW, I now use instance profile credentials pretty regularly for
my test runs of all patches I review.
With the rebase and addressing the above, all feedback has been resolved. I
plan to commit based on Steve's +1 after a fresh pre-commit run.
> S3A: Reduce high number of connections to EC2 Instance Metadata Service
> caused by InstanceProfileCredentialsProvider.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13727
> URL: https://issues.apache.org/jira/browse/HADOOP-13727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Assignee: Chris Nauroth
> Priority: Minor
> Attachments: HADOOP-13727-branch-2.001.patch,
> HADOOP-13727-branch-2.002.patch, HADOOP-13727-branch-2.003.patch,
> HADOOP-13727-branch-2.004.patch, HADOOP-13727-branch-2.005.patch
>
>
> When running in an EC2 VM, S3A can make use of
> {{InstanceProfileCredentialsProvider}} from the AWS SDK to obtain credentials
> from the EC2 Instance Metadata Service. We have observed that for a highly
> multi-threaded application, this may generate a high number of calls to the
> Instance Metadata Service. The service may throttle the client by replying
> with an HTTP 429 response or forcibly closing connections. We can greatly
> reduce the number of calls to the service by enforcing that all threads use a
> single shared instance of {{InstanceProfileCredentialsProvider}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
