Hi Einar, How did you put the data onto S3, using Hadoop's S3 FileSystem or using other S3 tools? If it's the latter then it won't work as the s3 scheme is for Hadoop's block-based S3 storage. Native S3 support is coming - see https://issues.apache.org/jira/browse/HADOOP-930, but it's not integrated yet.
Tom On Thu, May 29, 2008 at 10:15 PM, Einar Vollset <[EMAIL PROTECTED]> wrote: > Hi, > > I'm using the current Hadoop ec2 image (ami-ee53b687), and am having > some trouble getting hadoop > to access S3. Specifically, I'm trying to copy files from my bucket, > into HDFS on the running cluster, so > (on the master on the booted cluster) I do: > > hadoop-0.17.0 einar$ bin/hadoop distcp > s3://ID:[EMAIL PROTECTED]/ input > 08/05/29 14:10:44 INFO util.CopyFiles: srcPaths=[ > s3://ID:[EMAIL PROTECTED]/] > 08/05/29 14:10:44 INFO util.CopyFiles: destPath=input > 08/05/29 14:10:46 WARN fs.FileSystem: "localhost:9000" is a deprecated > filesystem name. Use "hdfs://localhost:9000/" instead. > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input > source s3://ID:[EMAIL PROTECTED]/ does not > exist. > at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578) > at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) > at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) > > ..which clearly doesn't work. The ID:SECRET are right - as if I change > them I get : > > org.jets3t.service.S3ServiceException: S3 HEAD request failed. > ResponseCode=403, ResponseMessage=Forbidden > ..etc > > I suspect it might be a generic problem, as if I do: > > bin/hadoop fs -ls s3://ID:[EMAIL PROTECTED]/ > > I get: > ls: Cannot access s3://ID:[EMAIL PROTECTED]/ : > No such file or directory. > > > ..even though the bucket is there and has a lot of data in it. > > > Any thoughts? > > Cheers, > > Einar >
