[ 
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124573#comment-16124573
 ] 

Steve Loughran commented on HADOOP-14749:
-----------------------------------------

# We haven't any tests of IAM role with r/o bucket and r/w of DDB. I did 
accidentally try to use s3guard and landsat, and don't remember it being a 
particularly good experience. That's really something we need to test before 
making any statement about its use. Hence my caution. I will state "this has 
been undertested and may not work"
 
# I already have throttling in the fault injection in HADOOP-13786., 
[here|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java].
 Shows how we don't handle it at all well today. It's not an optimistic promise 
then, more a "forward looking on a future merge of existing code" statement. 
The hard part isn't this injection, it's reworking every since direct call of 
the AWS SDK (other than via xfer manager) to handle the error. Hence 
[S3ALambda|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ALambda.java],
 which I use for all the committer and upload calls in 
[WriteOperationHelper|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java].
 We will need to go through every single API GET/HEAD call the same way for 
resilience.
 

> review s3guard docs & code prior to merge
> -----------------------------------------
>
>                 Key: HADOOP-14749
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14749
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation, fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14749-HADOOP-13345-001.patch, 
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch, 
> HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence 
> the changes)
> * Review the new tests



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to