[
https://issues.apache.org/jira/browse/HADOOP-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222597#comment-13222597
]
Jagane Sundar commented on HADOOP-5836:
---------------------------------------
A note on running the unit test for this (and possibly other s3/s3n driver
patches):
The name of the unit test is Jets3tNativeS3FileSystemContractTest and it is run
using the command:
$ ant -Dtestcase=Jets3tNativeS3FileSystemContractTest test-core
To run this unit test, you need to add your own S3 credentials to the file
src/test/core-site.xml. Edit src/test/core-site.xml and add details of a test
s3 and a test s3n filesystems for the test, as shown below. In this example,
you need to replace AAAAAAAAAAAAAAAAAAA:bbbbbbbbbbbbbbbbbbbbb with urlencoded
versions of your S3 access key id, and secret access key. Use the program
urlencode on linux to urlencode.
<property>
<name>test.fs.s3n.name</name>
<value>s3n://AAAAAAAAAAAAAAAAAAA:bbbbbbbbbbbbbbbbbbbbb@yourbucketname</value>
<description>The name of the s3n file system for testing.</description>
</property>
<property>
<name>test.fs.s3.name</name>
<value>s3://AAAAAAAAAAAAAAAAAAA:bbbbbbbbbbbbbbbbbbbbb@yourbucketname</value>
<description>The name of the s3 file system for testing.</description>
</property>
> Bug in S3N handling of directory markers using an object with a trailing "/"
> causes jobs to fail
> ------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5836
> URL: https://issues.apache.org/jira/browse/HADOOP-5836
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.18.3
> Reporter: Ian Nowland
> Assignee: Ian Nowland
> Fix For: 0.21.0
>
> Attachments: HADOOP-5836-0-for_branch_1_0.patch, HADOOP-5836-0.patch,
> HADOOP-5836-2.patch
>
>
> Some tools which upload to S3 and use a object terminated with a "/" as a
> directory marker, for instance "s3n://mybucket/mydir/". If asked to iterate
> that "directory" via listStatus(), then the current code will return an empty
> file "", which the InputFormatter happily assigns to a split, and which later
> causes a task to fail, and probably the job to fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
