[
https://issues.apache.org/jira/browse/HADOOP-19181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899116#comment-17899116
]
Steve Loughran commented on HADOOP-19181:
-----------------------------------------
update: nothing appears to be happening in the SDK. Even though it is a one
line fix -change the token refresh time to 60s before they expire. This is what
we do ourselves.
Because it's so trivial I don't see the point in as implementing a workaround.
This will take a few days work and ongoing maintenance.
Instead I am using this issue as an experiment to see if the SDK team actually
respond to issues.. It very surprising that something which spans all
non-container deployments within AWS infrastructure and is a trivial fix does
not seem to get any attention whatsoever. I believe our processes are in fact
more responseive than this, even though their SDK has a two week release cycle.
The STK box also leaks information about their test process. The
{{ContainerCredentialsProvider}} does not have this bug as it refreshes in a
timely manner. The implication is that dev/test is taking place in containers,
rather than raw VMs. Alternatively: whatever load tests they are doing don't
include credential renewal.
[~ahmar] [~srahman]
> S3A) IAMCredentialsProvider throttling results in AWS auth failures
> -------------------------------------------------------------------
>
> Key: HADOOP-19181
> URL: https://issues.apache.org/jira/browse/HADOOP-19181
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Priority: Major
>
> Tests report throttling errors in IAM being remapped to noauth and failure
> Again, impala tests, but with multiple processes on same host. this means
> that HADOOP-18945 isn't sufficient as even if it ensures a singleton instance
> for a process
> * it doesn't if there are many test buckets (fixable)
> * it doesn't work across processes (not fixable)
> we may be able to
> * use a singleton across all filesystem instances
> * once we know how throttling is reported, handle it through retries +
> error/stats collection
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
