[
http://issues.apache.org/jira/browse/HADOOP-574?page=comments#action_12453453 ]
Tom White commented on HADOOP-574:
----------------------------------
As an experiment I modified FileSystem.getNamed() to include a check for s3
(note that this change is not in HADOOP-574-v2.patch):
if ("local".equals(name)) {
fs = new LocalFileSystem(conf);
} else if ("s3".equals(name)) {
fs = new S3FileSystem(conf);
} else {
fs = new DistributedFileSystem(DataNode.createSocketAddr(name), conf);
}
I then built a hadoop distribution, unpacked, created a hadoop-site.xml wih my
S3 keys and bucket name and ran the following in the bin directory:
./hadoop dfs -conf hadoop-site.xml -fs s3 -mkdir /tmp/tom
./hadoop dfs -conf hadoop-site.xml -fs s3 -copyFromLocal s3test.txt
/tmp/tom/s3test.txt
./hadoop dfs -conf hadoop-site.xml -fs s3 -copyToLocal /tmp/tom/s3test.txt
s3test.copy.txt
diff s3test.copy.txt s3test.txt
./hadoop dfs -conf hadoop-site.xml -fs s3 -rm /tmp/tom/s3test.txt
./hadoop dfs -conf hadoop-site.xml -fs s3 -rmr /tmp/tom
All the commands succeeded and the diff showed that the files were the same.
This was a great sanity check!
This suggests to me that DFSShell is really more general than DFS - perhaps it
should be renamed FSShell?
Next steps - I guess I need to see how the S3 implementation fits with
HADOOP-571. Suggestions welcome.
> want FileSystem implementation for Amazon S3
> --------------------------------------------
>
> Key: HADOOP-574
> URL: http://issues.apache.org/jira/browse/HADOOP-574
> Project: Hadoop
> Issue Type: New Feature
> Components: fs
> Affects Versions: 0.9.0
> Reporter: Doug Cutting
> Attachments: dependencies.zip, HADOOP-574-v2.patch, HADOOP-574.patch
>
>
> An S3-based Hadoop FileSystem would make a great addition to Hadoop.
> It would facillitate use of Hadoop on Amazon's EC2 computing grid, as
> discussed here:
> http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg00318.html
> This is related to HADOOP-571, which would make Hadoop's FileSystem
> considerably easier to extend.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
