Ilya Fourmanov created HADOOP-14620:
---------------------------------------
Summary: S3A authentication failure for regions other than
us-east-1
Key: HADOOP-14620
URL: https://issues.apache.org/jira/browse/HADOOP-14620
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 2.7.3, 2.8.0
Reporter: Ilya Fourmanov
hadoop fs s3a:// operations fail authentication for s3 buckets hosted in
regions other than default us-east-1
Steps to reproduce:
# create s3 bucket in eu-west-1
# Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run
following command:
{code}
hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls
s3a://your-eu-west-1-hosted-bucket/
{code}
Expected behaviour:
You will see listing of the bucket
Actual behaviour:
You will get 403 Authentication Denied response for AWS S3.
Reason is mismatch in string to sign as defined in
http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html provided
by hadoop and expected by AWS.
If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes
returned by AWS, you will see that AWS expects CanonicalizedResource to be in
form
/your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/.
Hadoop provides it as /your-eu-west-1-hosted-bucket/
Note that AWS documentation doesn't explicitly state that endpoint or full dns
address should be appended to CanonicalizedResource however practice shows it
is actually required.
I've also submitted this to AWS for them to correct behaviour or documentation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
