>From the log, I guess that you have made wrong configurations or operations. The S3 storage layer is at the same level with HDFS. So if you configure to use S3, it's irrational to occur the HDFS related logs such as "org.apache.hadoop.hdfs.server.namenode.FSNamesystem". The HDFS related logs only occur at the condition that you use HDFS as the underlying storage layer.
I infer that you should not format the file system, because S3 is already service delivery and the Hadoop framework just use it as the underlying file system. Don't execute " bin/hadoop namenode -format ", it is used only when your underlying file system is HDFS. Yanbo 2012/7/25 Alok Kumar <alok...@gmail.com> > Hi Yanbo, > > I'm using hadoop-1.0.1 on local machine (not EC2) > > conf/master : localhost > conf/slave : localhost > conf/core-site.xml : > <property> > <name>fs.default.name</name> > <value>s3://bucketname</value> > </property> > > <property> > <name>fs.s3.awsAccessKeyId</name> > <value>***********</value> > </property> > > <property> > <name>fs.s3.awsSecretAccessKey</name> > <value>**************</value> > </property> > > /etc/hosts : > 192.168.2.24 DRPM4 # Added by NetworkManager > 127.0.0.1 localhost.localdomain localhost > > > $ bin/hadoop namenode -format : successful > > $ bin/start-dfs.sh > > Namenode log : > > 2012-07-25 13:19:32,896 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: > /************************************************************ > STARTUP_MSG: Starting NameNode > STARTUP_MSG: host = DRPM4/192.168.2.24 > STARTUP_MSG: args = [] > STARTUP_MSG: version = 1.0.1 > STARTUP_MSG: build = > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r > 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 > ************************************************************/ > 2012-07-25 13:19:33,037 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from > hadoop-metrics2.properties > 2012-07-25 13:19:33,049 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > MetricsSystem,sub=Stats registered. > 2012-07-25 13:19:33,050 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > period at 10 second(s). > 2012-07-25 13:19:33,050 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system > started > 2012-07-25 13:19:33,889 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi > registered. > 2012-07-25 13:19:33,896 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already > exists! > 2012-07-25 13:19:33,904 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm > registered. > 2012-07-25 13:19:33,906 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > NameNode registered. > 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: VM > type = 32-bit > 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: 2% max > memory = 17.77875 MB > 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: > capacity = 2^22 = 4194304 entries > 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: > recommended=4194304, actual=4194304 > 2012-07-25 13:19:33,986 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=alok > 2012-07-25 13:19:33,986 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup > 2012-07-25 13:19:33,986 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isPermissionEnabled=true > 2012-07-25 13:19:33,995 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.block.invalidate.limit=100 > 2012-07-25 13:19:33,995 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), > accessTokenLifetime=0 min(s) > 2012-07-25 13:19:34,396 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered > FSNamesystemStateMBean and NameNodeMXBean > 2012-07-25 13:19:34,416 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names > occuring more than 10 times > 2012-07-25 13:19:34,442 INFO org.apache.hadoop.hdfs.server.common.Storage: > Number of files = 1 > 2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage: > Number of files under construction = 0 > 2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage: > Image file of size 110 loaded in 0 seconds. > 2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage: > Edits file /home/alok/work/s3s/hadoop/namenode/current/edits of size 4 > edits # 0 loaded in 0 seconds. > 2012-07-25 13:19:34,451 INFO org.apache.hadoop.hdfs.server.common.Storage: > Image file of size 110 saved in 0 seconds. > 2012-07-25 13:19:35,161 INFO org.apache.hadoop.hdfs.server.common.Storage: > Image file of size 110 saved in 0 seconds. > 2012-07-25 13:19:35,873 INFO > org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 > entries 0 lookups > 2012-07-25 13:19:35,873 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading > FSImage in 1900 msecs > 2012-07-25 13:19:35,888 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks > = 0 > 2012-07-25 13:19:35,888 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid > blocks = 0 > 2012-07-25 13:19:35,888 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > under-replicated blocks = 0 > 2012-07-25 13:19:35,888 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > over-replicated blocks = 0 > 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE* > Safe mode termination scan for invalid, over- and under-replicated blocks > completed in 15 msec > 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE* > Leaving safe mode after 1 secs. > 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE* > Network topology has 0 racks and 0 datanodes > 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE* > UnderReplicatedBlocks has 0 blocks > 2012-07-25 13:19:35,897 INFO org.apache.hadoop.util.HostsFileReader: > Refreshing hosts (include/exclude) list > 2012-07-25 13:19:35,905 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue > QueueProcessingStatistics: First cycle completed 0 blocks in 7 msec > 2012-07-25 13:19:35,905 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue > QueueProcessingStatistics: Queue flush completed 0 blocks in 7 msec > processing time, 7 msec clock time, 1 cycles > 2012-07-25 13:19:35,905 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > FSNamesystemMetrics registered. > 2012-07-25 13:19:35,906 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue > QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec > 2012-07-25 13:19:35,906 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue > QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec > processing time, 1 msec clock time, 1 cycles > 2012-07-25 13:19:35,916 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > transactions: 0 Total time for transactions(ms): 0Number of transactions > batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 > 2012-07-25 13:19:35,916 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicationMonitor > thread received InterruptedException.java.lang.InterruptedException: sleep > interrupted > 2012-07-25 13:19:35,917 INFO > org.apache.hadoop.hdfs.server.namenode.DecommissionManager: Interrupted > Monitor > java.lang.InterruptedException: sleep interrupted > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65) > at java.lang.Thread.run(Thread.java:662) > 2012-07-25 13:19:35,993 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException: > Problem binding to *bucket*/67.215.65.132:8020 : Cannot assign requested > address > at org.apache.hadoop.ipc.Server.bind(Server.java:227) > at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301) > at org.apache.hadoop.ipc.Server.<init>(Server.java:1483) > at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:294) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) > Caused by: java.net.BindException: Cannot assign requested address > at sun.nio.ch.Net.bind(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at org.apache.hadoop.ipc.Server.bind(Server.java:225) > ... 8 more > > 2012-07-25 13:19:35,994 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at DRPM4/192.168.2.24 > ************************************************************/ > > Thanks, > > > > On Wed, Jul 25, 2012 at 12:05 PM, Yanbo Liang <yanboha...@gmail.com>wrote: > >> Could you provide your execute environment, operation procedure and the >> detail log information? >> >> >> 2012/7/24 Alok Kumar <alok...@gmail.com> >> >>> Hi Yanbo, >>> >>> Thank you for your reply.. >>> >>> Now I've made changes exactly , present in this link >>> http://wiki.apache.org/hadoop/AmazonS3 >>> >>> But my namenode is not coming up with exception (I tried it both locally >>> and inside EC2) >>> >>> ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: >>> java.net.BindException: Problem binding to <bucket-name>. >>> s3.amazonaws.com/207.171.163.14:8020 : Cannot assign requested address >>> >>> fs.default.name = s3://<mybucket> // Error : UnknownHost Exception >>> in Namenode! >>> or >>> fs.default.name = s3://<mybucket>.s3.amazonaws.com // Error >>> BindException in Namenode log! >>> >>> Also I can't see any dir created inside my bucket. ( I'm able to execute >>> command $ bin/hadoop dfs -ls s3://<bucket>/ ) >>> >>> "bin/hadoop namenode -format " is saying succesfully formatted namenode >>> dir S3://bucket/hadoop/namenode , when it is not even existing there! >>> >>> >>> any suggestion? >>> >>> Thanks again. >>> >>> >>> On Tue, Jul 24, 2012 at 4:11 PM, Yanbo Liang <yanboha...@gmail.com>wrote: >>> >>>> I think you have made confusion about the integration of hadoop and S3. >>>> 1) If you set "dfs.data.dir=s3://******", it means that you set S3 as >>>> the DataNode local storage. >>>> You are still using HDFS as the under storage layer. >>>> As far as I know, it is not supported at present. >>>> 2) The right way of using S3 integrated with Hadoop is to replace the >>>> HDFS with S3 just like you have tried. >>>> But I think you have missed some configuration parameters. >>>> " fs.default.name=s3://<<mybucket> " is the most important parameter >>>> when you are using S3 to replace HDFS, but it is not enough. The detail >>>> configuration can be obtained from here >>>> http://wiki.apache.org/hadoop/AmazonS3 >>>> >>>> Yanbo >>>> >>>> >>>> 2012/7/23 Alok Kumar <alok...@gmail.com> >>>> >>>>> Hello Group, >>>>> >>>>> I've hadoop setup locally running. >>>>> >>>>> Now I want to use Amazon s3://<mybucket> as my data store, >>>>> so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my >>>>> hdfs-site.xml, Is it the correct way? >>>>> I'm getting error : >>>>> >>>>> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid >>>>> directory in dfs.data.dir: can not create directory: >>>>> s3://<mybucket>/hadoop >>>>> 2012-07-23 13:15:06,260 ERROR >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in >>>>> dfs.data.dir are invalid. >>>>> >>>>> and >>>>> when i changed like " dfs.data.dir=s3://<mybucket>/ " >>>>> I got error : >>>>> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: >>>>> java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected: >>>>> file:/// >>>>> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381) >>>>> at >>>>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55) >>>>> at >>>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393) >>>>> at >>>>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >>>>> at >>>>> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146) >>>>> at >>>>> org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682) >>>>> >>>>> Also, >>>>> When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not >>>>> coming up with error : ERROR >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException: >>>>> (Any way I want to run namenode locally, so I reverted it back to >>>>> hdfs://localhost:9000 ) >>>>> >>>>> Your help is highly appreciated! >>>>> Thanks >>>>> -- >>>>> Alok Kumar >>>>> >>>> >>>> >>> >>> >>> -- >>> Alok Kumar >>> >>> >>> >> > > > -- > Alok Kumar > > >
