That wiki page mentiones hadoop-site.xml, but this is old, now you have core-site.xml and hdfs-site.xml, so which one do you put it in?
Thank you (and good night Central Time:) mark On Wed, Jan 18, 2012 at 12:52 AM, Harsh J <[email protected]> wrote: > When using S3 you do not need to run any component of HDFS at all. It > is meant to be an alternate FS choice. You need to run only MR. > > The wiki page at http://wiki.apache.org/hadoop/AmazonS3 mentions on > how to go about specifying your auth details to S3, either directly > via the fs.default.name URI or via the additional properties > fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey. Does this not work > for you? > > On Wed, Jan 18, 2012 at 12:14 PM, Mark Kerzner <[email protected]> > wrote: > > Well, here is my error message > > > > Starting Hadoop namenode daemon: starting namenode, logging to > > /usr/lib/hadoop-0.20/logs/hadoop-hadoop-namenode-ip-10-126-11-26.out > > ERROR. Could not start Hadoop namenode daemon > > Starting Hadoop secondarynamenode daemon: starting secondarynamenode, > > logging to > > > /usr/lib/hadoop-0.20/logs/hadoop-hadoop-secondarynamenode-ip-10-126-11-26.out > > Exception in thread "main" java.lang.IllegalArgumentException: Invalid > URI > > for NameNode address (check fs.default.name): s3n://myname.testdata is > not > > of scheme 'hdfs'. > > at > > > org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:224) > > at > > > org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress(NameNode.java:209) > > at > > > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:182) > > at > > > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:150) > > at > > > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:624) > > ERROR. Could not start Hadoop secondarynamenode daemon > > > > And, if I don't need to start the NameNode, then where do I give the S3 > > credentials? > > > > Thank you, > > Mark > > > > > > On Wed, Jan 18, 2012 at 12:36 AM, Harsh J <[email protected]> wrote: > > > >> Hey Mark, > >> > >> What is the exact trouble you run into? What do the error messages > >> indicate? > >> > >> This should be definitive enough I think: > >> http://wiki.apache.org/hadoop/AmazonS3 > >> > >> On Wed, Jan 18, 2012 at 11:55 AM, Mark Kerzner < > [email protected]> > >> wrote: > >> > Hi, > >> > > >> > whatever I do, I can't make it work, that is, I cannot use > >> > > >> > s3://host > >> > > >> > or s3n://host > >> > > >> > as a replacement for HDFS while runnings EC2 cluster. I change the > >> settings > >> > in the core-file.xml, in hdfs-site.xml, and start hadoop services, > and it > >> > fails with error messages. > >> > > >> > Is there a place where this is clearly described? > >> > > >> > Thank you so much. > >> > > >> > Mark > >> > >> > >> > >> -- > >> Harsh J > >> Customer Ops. Engineer, Cloudera > >> > > > > -- > Harsh J > Customer Ops. Engineer, Cloudera >
