It's random. On Jul 4, 2013 3:33 PM, "Bing Jiang" <[email protected]> wrote:
> If not set cluster id in formatting the Namenode, is there a policy in > hdfs to guarantee the even of distributing DataNodes into different > Namespace, or just randomly? > > > > 2013/7/4 Azuryy Yu <[email protected]> > >> Additional, >> >> If these are two new clusters, then on each namenode, using "hdfs >> namenode -format -clusterID yourID" >> >> But if you want to upgrade these two clusters from NonHA to HA, then >> using "bin/start-dfs.sh -upgrade -clusterID yourID" >> >> >> >> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <[email protected]> wrote: >> >>> >>> This is because you don't use the same clusterID. all data nodes and >>> namenodes should use the same clusterID. >>> >>> >>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <[email protected]>wrote: >>> >>>> Hi, all >>>> >>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for >>>> hbase cluster, and the other one is for common use.At the same time, we use >>>> Quorum Journal policy as HA. >>>> >>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace >>>> >>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace. >>>> >>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode >>>> >>>> After launching the hdfs cluster all, there is something which makes >>>> me confused, that each namespace has half of the datanodes. >>>> >>>> NameNode 'GS-CIX-SEV0004:9100' >>>> >>>> Started: Thu Jul 04 10:28:00 CST 2013 >>>> Version: 2.0.5-alpha, 1488459 >>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha >>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7 >>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940 >>>> >>>> >>>> Browse the filesystem >>>> NameNode Logs >>>> Go back to DFS home >>>> >>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ >>>> Live Datanodes : 4 >>>> >>>> >>>> Last Admin Configured Used Non DFS Remaining >>>> Used Used Remaining Block Block Pool Failed >>>> Node Contact State Capacity (GB) Used (GB) >>>> (%) (%) (%) Blocks Pool Used (%)> Volumes >>>> (GB) >>>> (GB) Used (GB) Blocks >>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03 >>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57 >>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46 >>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77 >>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> >>>> >>>> Another Namespace's NameNode: >>>> >>>> NameNode 'GS-CIX-SEV0001:9100' >>>> >>>> Started: Thu Jul 04 10:19:03 CST 2013 >>>> Version: 2.0.5-alpha, 1488459 >>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha >>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6 >>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309 >>>> >>>> >>>> Browse the filesystem >>>> NameNode Logs >>>> Go back to DFS home >>>> >>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ >>>> Live Datanodes : 4 >>>> >>>> >>>> Last Admin Configured Used Non DFS Remaining >>>> Used Used Remaining Block Block Pool Failed >>>> Node Contact State Capacity (GB) Used (GB) >>>> (%) (%) (%) Blocks Pool Used (%)> Volumes >>>> (GB) >>>> (GB) Used (GB) Blocks >>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53 >>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85 >>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16 >>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0 >>>> >>>> └────────────┘ >>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82 >>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0 >>>> >>>> >>>> >>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this: >>>> 2013-07-04 10:34:51,699 FATAL >>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for >>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id >>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/ >>>> 10.100.2.1:9100 >>>> java.io.IOException: Inconsistent storage IDs. Name-node returned >>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690 >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664) >>>> at java.lang.Thread.run(Thread.java:662) >>>> >>>> It is proved that one datanode has been required to attached to only >>>> one namespace? >>>> >>>> Any views about it will be thankful. >>>> >>>> Regards~ >>>> >>>> >>>> -- >>>> Bing Jiang >>>> Tel：(86)134-2619-1361 >>>> weibo: http://weibo.com/jiangbinglover >>>> BLOG: http://blog.sina.com.cn/jiangbinglover >>>> National Research Center for Intelligent Computing Systems >>>> Institute of Computing technology >>>> Graduate University of Chinese Academy of Science >>>> >>> >>> >> > > > -- > Bing Jiang > Tel：(86)134-2619-1361 > weibo: http://weibo.com/jiangbinglover > BLOG: http://blog.sina.com.cn/jiangbinglover > National Research Center for Intelligent Computing Systems > Institute of Computing technology > Graduate University of Chinese Academy of Science >
