[
https://issues.apache.org/jira/browse/HADOOP-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649837#action_12649837
]
David Phillips commented on HADOOP-4422:
----------------------------------------
I ran the tests as follows after setting the correct test buckets and keys in
src/test/hadoop-site.xml:
ant -Dtestcase=Jets3tS3FileSystemContractTest test
ant -Dtestcase=Jets3tNativeS3FileSystemContractTest test
They seem to pass:
Testsuite: org.apache.hadoop.fs.s3.Jets3tS3FileSystemContractTest
Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 131.575 sec
Testsuite: org.apache.hadoop.fs.s3native.Jets3tNativeS3FileSystemContractTest
Tests run: 26, Failures: 0, Errors: 0, Time elapsed: 52.694 sec
However, they both produce hundreds of warnings:
(s3) 2008-11-21 15:51:28,800 WARN httpclient.RestS3Service
(RestS3Service.java:performRequest(317)) - Response '/%2Ftest' - Unexpected
response code 404, expected 200
(s3n) 2008-11-21 15:34:55,646 WARN httpclient.RestS3Service
(RestS3Service.java:performRequest(317)) - Response '/test' - Unexpected
response code 404, expected 200
Any ideas?
> S3 native fs should not create bucket
> -------------------------------------
>
> Key: HADOOP-4422
> URL: https://issues.apache.org/jira/browse/HADOOP-4422
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.18.1
> Reporter: David Phillips
> Assignee: David Phillips
> Attachments: hadoop-s3n-nocreate.patch, hadoop-s3n-nocreate.patch
>
>
> S3 native file system tries to create the bucket at every initialization.
> This is bad because
> * Every S3 operation costs money. These unnecessary calls are an unnecessary
> expense.
> * These calls can fail when called concurrently. This makes the file system
> unusable in large jobs.
> * Any operation, such as a "fs -ls", creates a bucket. This is
> counter-intuitive and undesirable.
> The initialization code should assume the bucket exists:
> * Creating a bucket is a very rare operation. Accounts are limited to 100
> buckets.
> * Any check at initialization for bucket existence is a waste of money.
> Per Amazon: "Because bucket operations work against a centralized, global
> resource space, it is not appropriate to make bucket create or delete calls
> on the high availability code path of your application. It is better to
> create or delete buckets in a separate initialization or setup routine that
> you run less often."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
