[
https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337042#comment-15337042
]
Chris Nauroth edited comment on HADOOP-13287 at 6/17/16 9:49 PM:
-----------------------------------------------------------------
My test run was on branch-2.8 against an S3 bucket in US-west-2. What I saw
happening was a double decoding in {{S3xLoginHelper#extractLoginDetails}}:
{code}
public static Login extractLoginDetails(URI name) {
try {
String authority = name.getAuthority();
...
String password = URLDecoder.decode(login.substring(loginSplit + 1),
"UTF-8");
{code}
According to the JavaDocs for
[{{URI#getAuthority}}|http://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getAuthority()],
it performs decoding already on the output. Then we do a second explicit
decoding by calling {{URLDecoder#decode}}. First, {{getAuthority}} translates
"%2B" to "\+". Then, {{URLDecoder#decode}} translates "\+" to " ", which isn't
correct for the credentials.
However, this appear to be only a problem in the JUnit test runs. I also built
a distro and tested manually with URIs that contain '\+' encoded as "%2B", and
that worked just fine. The reason it works fine there is because of different
encoding rules applied by round-tripping through a {{Path}} before the
{{FileSystem#get}} call gets triggered. With {{Path}}, the '\+' gets
double-encoded to "%252B", so double-decoding at the S3A layer is correct logic.
To make this work, the test should follow the same encoding as would be used on
the CLI. The attached path switches from constructing a {{URI}} to
constructing a {{Path}}. I switched the exception stifling logic to catch
{{IllegalArgumentException}}, becauses that's what {{Path}} throws. With this,
the test passes with a secret containing a '+'. [[email protected]] or
[~raviprak], I understand one of you might have a secret with a '/' from your
work on HADOOP-3733. Would you mind testing this patch to make sure the test
still passes with '/'?
was (Author: cnauroth):
My test run was on branch-2.8 against an S3 bucket in US-west-2. What I saw
happening was a double decoding in {{S3xLoginHelper#extractLoginDetails}}:
{code}
public static Login extractLoginDetails(URI name) {
try {
String authority = name.getAuthority();
...
String password = URLDecoder.decode(login.substring(loginSplit + 1),
"UTF-8");
{code}
According to the JavaDocs for
[{{URI#getAuthority}}|http://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getAuthority()],
it performs decoding already on the output. Then we do a second explicit
decoding by calling {{URLDecoder#decode}}. First, {{getAuthority}} translates
"%2B" to "\+". Then, {{URLDecoder#decode}} translates "\+" to " ", which isn't
correct for the credentials.
However, this appear to be only a problem in the JUnit test runs. I also built
a distro and tested manually with URIs that contain '+' encoded as "%2B", and
that worked just fine. The reason it works fine there is because of different
encoding rules applied by round-tripping through a {{Path}} before the
{{FileSystem#get}} call gets triggered. With {{Path}}, the '+' gets
double-encoded to "%252B", so double-decoding at the S3A layer is correct logic.
To make this work, the test should follow the same encoding as would be used on
the CLI. The attached path switches from constructing a {{URI}} to
constructing a {{Path}}. I switched the exception stifling logic to catch
{{IllegalArgumentException}}, becauses that's what {{Path}} throws. With this,
the test passes with a secret containing a '+'. [[email protected]] or
[~raviprak], I understand one of you might have a secret with a '/' from your
work on HADOOP-3733. Would you mind testing this patch to make sure the test
still passes with '/'?
> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains
> '+'.
> -------------------------------------------------------------------------------
>
> Key: HADOOP-13287
> URL: https://issues.apache.org/jira/browse/HADOOP-13287
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
> Attachments: HADOOP-13287.001.patch
>
>
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an
> AWS secret key containing a '/'. The patch added a new test suite:
> {{TestS3ACredentialsInURL}}. One of the tests fails if your AWS secret key
> contains a '+'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
