[ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-3733: ----------------------------------- Attachment: HADOOP-3733-branch-2-003.patch Patch 003 * fixes canonicalization so that there shouldn't be errors if path checking now that auth details are being stripped out of fsUri * tests this * I've not been able to replicate the checkpath/canonicalization problem which was reported to me by Ravi; he'll have to test himself. * special message for the case where getHost()==null, getAuthority!=null; this situation arises if there is an unencoded / in the path: {code} -ls: Fatal internal error java.lang.NullPointerException: null uri host. This can be caused by unencoded / in the password string at java.util.Objects.requireNonNull(Objects.java:228) at org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:53) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:199) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2793) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:101) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:373) {code} > "s3:" URLs break when Secret Key contains a slash, even if encoded > ------------------------------------------------------------------ > > Key: HADOOP-3733 > URL: https://issues.apache.org/jira/browse/HADOOP-3733 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 0.17.1, 2.0.2-alpha > Reporter: Stuart Sierra > Assignee: Steve Loughran > Priority: Minor > Attachments: HADOOP-3733-20130223T011025Z.patch, > HADOOP-3733-branch-2-001.patch, HADOOP-3733-branch-2-002.patch, > HADOOP-3733-branch-2-003.patch, HADOOP-3733.patch, hadoop-3733.patch > > > When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, > distcp fails if the SECRET contains a slash, even when the slash is > URL-encoded as %2F. > Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH > And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > And your bucket is called "mybucket" > You can URL-encode the Secret KKey as > Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv > But this doesn't work: > {noformat} > $ bin/hadoop distcp file:///source > s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source] > 08/07/09 15:05:22 INFO util.CopyFiles: > destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest > 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: > mybucket > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > at > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339) > ... > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.fs.s3.S3Exception: > org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: > <?xml version="1.0" > encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The > request signature we calculated does not match the signature you provided. > Check your key and signing method.</Message> > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141) > ... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org