Indeed there was some subtlety that I did not understand But now I understand Thank you very much Vinithra
Antoine De : Vinithra Varadharajan [mailto:vinit...@cloudera.com] Envoyé : mercredi 25 juillet 2012 19:44 À : hdfs-user@hadoop.apache.org Objet : Re: HDFS Federation Cluster IDs not matched Antoine, In order to setup Federation, all the NameNodes have to be first formatted (or upgraded) with the same cluster ID. This is described here: http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html#Formatting_Namenodes. -Vinithra On Wed, Jul 25, 2012 at 5:01 AM, Antoine Boudot <abou...@cyres.fr<mailto:abou...@cyres.fr>> wrote: hello I'm beginner on technology hadoop and I encounter a problem , I am testing to set 2 namenode of 2 separate host with HDFS federation and a pool block is not to create to startup namenode log error here is the datanode 2012-07-25 11:20:39,818 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /couloir1/tmp/dfs/data is not formatted. 2012-07-25 11:20:39,818 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ... 2012-07-25 11:20:39,858 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled 2012-07-25 11:20:39,858 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /couloir1/tmp/dfs/data/current/BP-459972778-172.31.6.11-1343208016681 is not formatted. 2012-07-25 11:20:39,858 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ... 2012-07-25 11:20:39,858 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting block pool BP-459972778-172.31.6.11-1343208016681 directory /couloir1/tmp/dfs/data/current/BP-459972778-172.31.6.11-1343208016681/current 2012-07-25 11:20:39,860 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=91302135;bpid=BP-459972778-172.31.6.11-1343208016681;lv=-40;nsInfo=lv=-40;cid=CID-345eacf9-d084-4d24-84f0-8294ca7174d8;nsid=91302135;c=0;bpid=BP-459972778-172.31.6.11-1343208016681 2012-07-25 11:20:39,870 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /couloir1/tmp/dfs/data/current 2012-07-25 11:20:39,876 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean 2012-07-25 11:20:39,880 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1343224800880 with interval 21600000 2012-07-25 11:20:39,881 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-459972778-172.31.6.11-1343208016681 2012-07-25 11:20:39,894 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-459972778-172.31.6.11-1343208016681 (storage id DS-1163140142-172.31.6.10-50010-1343208039823) service to hadoop-n10/172.31.6.11:33333<http://172.31.6.11:33333> beginning handshake with NN 2012-07-25 11:20:39,899 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-321278605-172.31.4.59-1343207919699 (storage id DS-1163140142-172.31.6.10-50010-1343208039823) service to hadoop-vm03/172.31.4.59:33333<http://172.31.4.59:33333> java.io.IOException: Cluster IDs not matched: dn cid=CID-345eacf9-d084-4d24-84f0-8294ca7174d8 but ns cid=CID-9f7751ec-615b-4f3b-8644-5583ba263a0e; bpid=BP-321278605-172.31.4.59-1343207919699 at org.apache.hadoop.hdfs.server.datanode.DataNode.setClusterId(DataNode.java:307) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:801) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:308) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:660) at java.lang.Thread.run(Thread.java:662) 2012-07-25 11:20:39,899 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-321278605-172.31.4.59-1343207919699 (storage id DS-1163140142-172.31.6.10-50010-1343208039823) service to hadoop-vm03/172.31.4.59:33333<http://172.31.4.59:33333> 2012-07-25 11:20:39,900 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-321278605-172.31.4.59-1343207919699 (storage id DS-1163140142-172.31.6.10-50010-1343208039823) Here config for core-site.xml <property> <name>hadoop.tmp.dir</name> <value>/couloir1/tmp</value> </property> Here config for hdfs-site.xml <property> <name>dfs.federation.nameservices</name> <value>ns1,ns2</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.rpc-address.ns1</name> <value>hadoop-vm03:33333</value> </property> <property> <name>dfs.namenode.rpc-address.ns2</name> <value>hadoop-n10:33333</value> </property> </configuration> Help me please Thank you
