[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Fix Version/s: (was: 3.0.0-alpha1) (was: 2.9.0) 2.8.0 pulled the fix into Hadoop 2.8 as well; changed the "fixed for" version marker appropriately. > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733-branch-2-007.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-3733: - Resolution: Fixed Fix Version/s: 3.0.0-alpha1 2.9.0 Target Version/s: (was: 2.8.0) Status: Resolved (was: Patch Available) Thanks a lot everyone for your contributions on this long-standing issue. I'm glad we could close it out thanks to Steve! > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733-branch-2-007.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Patch Available (was: Open) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733-branch-2-007.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-007.patch Patch 007 Ravi's patch + javadocs. Tested against s3 ireland, and on the command line. It's notable that the secrets end up everywhere, such as in the output of the {{hadoop fs -ls}} command. Putting AWS login details in URLs is just wrong. > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733-branch-2-007.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Open (was: Patch Available) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Patch Available (was: Open) Patch 006; tested against S3 ireland. I haven't tested inline secrets in s3, s3n, or on the command line; just in a unit test that is set up to do this (and goes to effort to not log the details on a failure: the first time I've written a unit test to be deliberately useless when reporting failures) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-006.patch patch branch-2-006. Add a couple of lines in the documentation telling people not to put secrets in their URLs > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-005.patch Patch 0005 * all s3 filesystems warn that you shouldn't be putting secrets in your URLs * and s3n/s3 don't mention the technique in their error messages * javadocs * fix findbugs warnings, one through a fix, one through commenting it out. (it's essentially the same code copied and pasted from Filesystem; it's disabled there too). > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733-branch-2-005.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Open (was: Patch Available) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-004.patch patch branch-2-004; fixes checkstyle. Chris: thanks for the +1; I'm waiting for Ravi to do another attempt at trying to get this to work. FWIW, I don' think people should be trying to use credentials on the CLI. This patch tries to strip it from the URL and path, but they do creep out in error messages and stack traces. > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Patch Available (was: Open) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Open (was: Patch Available) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Open (was: Patch Available) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Patch Available (was: Open) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-003.patch Patch 003 * fixes canonicalization so that there shouldn't be errors if path checking now that auth details are being stripped out of fsUri * tests this * I've not been able to replicate the checkpath/canonicalization problem which was reported to me by Ravi; he'll have to test himself. * special message for the case where getHost()==null, getAuthority!=null; this situation arises if there is an unencoded / in the path: {code} -ls: Fatal internal error java.lang.NullPointerException: null uri host. This can be caused by unencoded / in the password string at java.util.Objects.requireNonNull(Objects.java:228) at org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:53) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:199) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2793) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:101) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:373) {code} > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-3733: -- Hadoop Flags: Reviewed +1 for the patch. I did a successful parallel test run against US-west-2. The Checkstyle warnings look trivial enough to fix on check-in if you'd like. Thank you for the patch. > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Patch Available (was: Open) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-002.patch > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Status: Open (was: Patch Available) > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Target Version/s: 2.8.0 (was: ) Status: Patch Available (was: Open) testing: S3 ireland. Intermittent failure of S3A root test (see HADOOP-13271), which goes away when rerun. > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.2-alpha, 0.17.1 >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3733: --- Attachment: HADOOP-3733-branch-2-001.patch Fix for this # pull out all URL user/pass extraction into a new class {{org.apache.hadoop.fs.s3native.S3xLoginHelper}}. It's in s3n as it is needed there too, I want it to outlive s3: removal, and reserve the option to backport. # this performs its own parsing of the user:pass from the authority info; handles / in passwords. # FS URI construction strips out the authority info, includes only the host in its URIs. # S3, S3N, S3A all use this codepath # S3A has had its code related to credentials modified to work with this too; it shares the same Login structure and is in a static S3AUtils method for easier testing. # Lots of unit tests to verify parsing works # There's an S3A functional test which verifies that passwords stuck in the FS URL are picked up. # I have tested that suite with a password with / in it If you do want to use a / in a password in a URL, do encode it with %2F; this will now be handled. > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-3733: - Target Version/s: 3.0.0 > "s3:" URLs break when Secret Key contains a slash, even if encoded > -- > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.17.1, 2.0.2-alpha >Reporter: Stuart Sierra >Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733.patch, > hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > encoding="UTF-8"?>SignatureDoesNotMatchThe > request signature we calculated does not match the signature you provided. > Check your key and signing method. > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-3733: - Status: Open (was: Patch Available) Cancelling patch since it no longer applies. s3: URLs break when Secret Key contains a slash, even if encoded -- Key: HADOOP-3733 URL: https://issues.apache.org/jira/browse/HADOOP-3733 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.0.2-alpha, 0.17.1 Reporter: Stuart Sierra Priority: Minor Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733.patch, hadoop-3733.patch When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F. Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv And your bucket is called mybucket You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv But this doesn't work: {noformat} $ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) ... With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: ?xml version=1.0 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe request signature we calculated does not match the signature you provided. Check your key and signing method./Message at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Chaiken updated HADOOP-3733: -- Affects Version/s: 2.0.2-alpha s3: URLs break when Secret Key contains a slash, even if encoded -- Key: HADOOP-3733 URL: https://issues.apache.org/jira/browse/HADOOP-3733 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 0.17.1, 2.0.2-alpha Reporter: Stuart Sierra Priority: Minor Attachments: hadoop-3733.patch When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F. Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv And your bucket is called mybucket You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv But this doesn't work: {noformat} $ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) ... With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: ?xml version=1.0 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe request signature we calculated does not match the signature you provided. Check your key and signing method./Message at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) ... {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Chaiken updated HADOOP-3733: -- Attachment: HADOOP-3733.patch patch for HADOOP-3733 on 2.0.2-alpha s3: URLs break when Secret Key contains a slash, even if encoded -- Key: HADOOP-3733 URL: https://issues.apache.org/jira/browse/HADOOP-3733 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 0.17.1, 2.0.2-alpha Reporter: Stuart Sierra Priority: Minor Attachments: hadoop-3733.patch, HADOOP-3733.patch When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F. Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv And your bucket is called mybucket You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv But this doesn't work: {noformat} $ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) ... With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: ?xml version=1.0 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe request signature we calculated does not match the signature you provided. Check your key and signing method./Message at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) ... {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Chaiken updated HADOOP-3733: -- Attachment: HADOOP-3733-20130223T011025Z.patch Added newly-required timeouts to the patch unit tests. (See HADOOP-9112.) s3: URLs break when Secret Key contains a slash, even if encoded -- Key: HADOOP-3733 URL: https://issues.apache.org/jira/browse/HADOOP-3733 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 0.17.1, 2.0.2-alpha Reporter: Stuart Sierra Priority: Minor Attachments: HADOOP-3733-20130223T011025Z.patch, hadoop-3733.patch, HADOOP-3733.patch When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F. Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv And your bucket is called mybucket You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv But this doesn't work: {noformat} $ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) ... With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: ?xml version=1.0 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe request signature we calculated does not match the signature you provided. Check your key and signing method./Message at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) ... {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded
[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Chaiken updated HADOOP-3733: -- Release Note: Allows userinfo component of URI authority to contain a slash (escaped as %2F). Especially useful for accessing AWS S3 with distcp or hadoop fs. Status: Patch Available (was: Open) s3: URLs break when Secret Key contains a slash, even if encoded -- Key: HADOOP-3733 URL: https://issues.apache.org/jira/browse/HADOOP-3733 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.0.2-alpha, 0.17.1 Reporter: Stuart Sierra Priority: Minor Attachments: HADOOP-3733-20130223T011025Z.patch, hadoop-3733.patch, HADOOP-3733.patch When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F. Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv And your bucket is called mybucket You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv But this doesn't work: {noformat} $ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) ... With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: ?xml version=1.0 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe request signature we calculated does not match the signature you provided. Check your key and signing method./Message at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) ... {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira