[jira] [Commented] (HADOOP-14620) S3A authentication failure for regions other than us-east-1
[ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074551#comment-16074551 ] Ilya Fourmanov commented on HADOOP-14620: - Upon further investigation it turned out that setting fs.s3a.bucket.dshbasebackup.endpoint=s3.eu-west-1.amazonaws.com seems to have no effect as hadoop was going through default endpoint s3.amazonaws.com. I'm on 2.7.3. However it turns out that using default endpoint actually works for buckets hosted in eu-west-1. And authentication succeeds for them. Going through region specific endpoint s3.eu-west-1.amazonaws.com fails with 403 > S3A authentication failure for regions other than us-east-1 > --- > > Key: HADOOP-14620 > URL: https://issues.apache.org/jira/browse/HADOOP-14620 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 2.7.3 >Reporter: Ilya Fourmanov > Attachments: s3-403.txt > > > hadoop fs s3a:// operations fail authentication for s3 buckets hosted in > regions other than default us-east-1 > Steps to reproduce: > # create s3 bucket in eu-west-1 > # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run > following command: > {code} > hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls > s3a://your-eu-west-1-hosted-bucket/ > {code} > Expected behaviour: > You will see listing of the bucket > Actual behaviour: > You will get 403 Authentication Denied response for AWS S3. > Reason is mismatch in string to sign as defined in > http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html > provided by hadoop and expected by AWS. > If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes > returned by AWS, you will see that AWS expects CanonicalizedResource to be in > form > /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/. > Hadoop provides it as /your-eu-west-1-hosted-bucket/ > Note that AWS documentation doesn't explicitly state that endpoint or full > dns address should be appended to CanonicalizedResource however practice > shows it is actually required. > I've also submitted this to AWS for them to correct behaviour or > documentation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14620) S3A authentication failure for regions other than us-east-1
[ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074461#comment-16074461 ] Ilya Fourmanov commented on HADOOP-14620: - That's extremely interesting. So {code} hadoop fs -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls s3a://dshbasebackup/ {code} fails for me with 403 as described above however if I use format as proposed by [~ste...@apache.org] {code} hadoop fs -D fs.s3a.bucket.dshbasebackup.endpoint=s3.eu-west-1.amazonaws.com -ls s3a://dshbasebackup/ {code} it works as expected. Now, what's the difference between those 2 formats? > S3A authentication failure for regions other than us-east-1 > --- > > Key: HADOOP-14620 > URL: https://issues.apache.org/jira/browse/HADOOP-14620 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 2.7.3 >Reporter: Ilya Fourmanov > Attachments: s3-403.txt > > > hadoop fs s3a:// operations fail authentication for s3 buckets hosted in > regions other than default us-east-1 > Steps to reproduce: > # create s3 bucket in eu-west-1 > # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run > following command: > {code} > hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls > s3a://your-eu-west-1-hosted-bucket/ > {code} > Expected behaviour: > You will see listing of the bucket > Actual behaviour: > You will get 403 Authentication Denied response for AWS S3. > Reason is mismatch in string to sign as defined in > http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html > provided by hadoop and expected by AWS. > If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes > returned by AWS, you will see that AWS expects CanonicalizedResource to be in > form > /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/. > Hadoop provides it as /your-eu-west-1-hosted-bucket/ > Note that AWS documentation doesn't explicitly state that endpoint or full > dns address should be appended to CanonicalizedResource however practice > shows it is actually required. > I've also submitted this to AWS for them to correct behaviour or > documentation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14620) S3A authentication failure for regions other than us-east-1
[ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Fourmanov updated HADOOP-14620: Attachment: s3-403.txt Attaching log file with 403 error. You can compare String to sign in debug output with string to sign returned by amazon s3 in HEX bytes. Decode using https://aws.amazon.com/code/199 > S3A authentication failure for regions other than us-east-1 > --- > > Key: HADOOP-14620 > URL: https://issues.apache.org/jira/browse/HADOOP-14620 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 2.7.3 >Reporter: Ilya Fourmanov > Attachments: s3-403.txt > > > hadoop fs s3a:// operations fail authentication for s3 buckets hosted in > regions other than default us-east-1 > Steps to reproduce: > # create s3 bucket in eu-west-1 > # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run > following command: > {code} > hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls > s3a://your-eu-west-1-hosted-bucket/ > {code} > Expected behaviour: > You will see listing of the bucket > Actual behaviour: > You will get 403 Authentication Denied response for AWS S3. > Reason is mismatch in string to sign as defined in > http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html > provided by hadoop and expected by AWS. > If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes > returned by AWS, you will see that AWS expects CanonicalizedResource to be in > form > /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/. > Hadoop provides it as /your-eu-west-1-hosted-bucket/ > Note that AWS documentation doesn't explicitly state that endpoint or full > dns address should be appended to CanonicalizedResource however practice > shows it is actually required. > I've also submitted this to AWS for them to correct behaviour or > documentation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14620) S3A authentication failure for regions other than us-east-1
Ilya Fourmanov created HADOOP-14620: --- Summary: S3A authentication failure for regions other than us-east-1 Key: HADOOP-14620 URL: https://issues.apache.org/jira/browse/HADOOP-14620 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.7.3, 2.8.0 Reporter: Ilya Fourmanov hadoop fs s3a:// operations fail authentication for s3 buckets hosted in regions other than default us-east-1 Steps to reproduce: # create s3 bucket in eu-west-1 # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run following command: {code} hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls s3a://your-eu-west-1-hosted-bucket/ {code} Expected behaviour: You will see listing of the bucket Actual behaviour: You will get 403 Authentication Denied response for AWS S3. Reason is mismatch in string to sign as defined in http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html provided by hadoop and expected by AWS. If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes returned by AWS, you will see that AWS expects CanonicalizedResource to be in form /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/. Hadoop provides it as /your-eu-west-1-hosted-bucket/ Note that AWS documentation doesn't explicitly state that endpoint or full dns address should be appended to CanonicalizedResource however practice shows it is actually required. I've also submitted this to AWS for them to correct behaviour or documentation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org