If you have two network cards and one hostname binding on each, please make sure to set the hostname you'd like to use in HBase in /etc/hosts on each nodes, including HMaster and every RS.
Hope this helps. Best Regards, Yu On 17 January 2017 at 23:47, Dima Spivak <[email protected]> wrote: > Is there any other DNS server running that might be confusing reverse > lookup? What happens if you run `host YOUR_RS_IP_ADDRESS`? > > And what kind of machines are you using in your deployment? > > Cheers, > > On Mon, Jan 16, 2017 at 11:34 PM C R <[email protected]> wrote: > > > Thanks, > > > > > > > > > > > > I deployed my HBase very simply, which has one Master and three > > regionservers. > > > > > > > > > > > > [hbase@bjsh19-16-30 conf]$ more regionservers > > > > bjsh19-16-33.qbos.com > > > > bjsh19-16-34.qbos.com > > > > bjsh19-16-35.qbos.com > > > > [hbase@bjsh19-16-30 conf]$ more hbase-site.xml > > > > > > > > ... > > > > > > > > <configuration> > > > > <property> > > > > <name>zookeeper.znode.parent</name> > > > > <value>/hbase117</value> > > > > </property> > > > > <property> > > > > <name>hbase.rootdir</name> > > > > <value>hdfs://bidc/hbase117</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.quorum</name> > > > > <value>bjsh19-16-30.qbos.com,bjsh19-16-31.qbos.com, > > bjsh19-16-32.qbos.com</value> > > > > </property> > > > > <property> > > > > <name>hbase.cluster.distributed</name> > > > > <value>true</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.clientPort</name> > > > > <value>2181</value> > > > > </property> > > > > </configuration> > > > > > > > > > > > > The special place is the file /etc/hosts with one ip mapping to two > > hostnames on all nodes,so it will have the message: > > > > > > > > ... > > > > > > > > the server that tried to transition was wjsa-tsl05,16020,1484623636195 > not > > the expected bjsh19-16-34.qbos.com,16020,1484623636195 > > > > > > > > ... > > > > > > > > > > > > ________________________________ > > > > 发件人: Dima Spivak <[email protected]> > > > > 发送时间: 2017年1月17日 4:50 > > > > 收件人: [email protected] > > > > 主题: Re: hbase has problems with two hostname > > > > > > > > Hi C R, > > > > > > > > Like many Hadoop-like services, HBase is pretty temperamental about > > > > requiring forward and reverse DNS to work properly. FWIW, the > configuration > > > > file where you can populate RegionServers doesn't tend to matter as long > as > > > > the hbase-site.xml file is populated correctly (it's just used to start > > > > daemons from one place). > > > > > > > > If you pass along more details about how exactly you're deploying HBase, > we > > > > might be able to give more advice. > > > > > > > > On Mon, Jan 16, 2017 at 8:00 PM C R <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more /etc/hosts > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > > > > 10.19.16.31 bjsh19-16-31.qbos.com wjsa-tsl02 > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There will have six regionservers listed in web console, but > > > > > > > > > > only three in the configuration file, metadata tables also are not > > online > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hmaster will be dead after a while. > > > > > > > > > > > > > > > what should I do? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > snapshot: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-01-17 11:45:24,394 INFO > > > > > [MASTER_SERVER_OPERATIONS-bjsh19-16-30:16000-0] > > master.AssignmentManager: > > > > > Assigning > > hbase:namespace,,1484623643279.30fab746cb3b6ceadcbda421459204b9. > > > > > to bjsh19-16-34.qbos.com,16020,1484623636195 > > > > > > > > > > > > > > > 2017-01-17 11:45:24,395 INFO [bjsh19-16-30:16000.activeMasterManager] > > > > > master.AssignmentManager: Joined the cluster in 23ms, failover=true > > > > > > > > > > > > > > > 2017-01-17 11:50:24,314 FATAL [bjsh19-16-30:16000.activeMasterManager] > > > > > master.HMaster: Failed to become active master > > > > > > > > > > > > > > > java.io.IOException: Timedout 300000ms waiting for namespace table to > be > > > > > assigned > > > > > > > > > > > > > > > at > > > > > > > org.apache.hadoop.hbase.master.TableNamespaceManager. > start(TableNamespaceManager.java:104) > > > > > > > > > > > > > > > at > > > > > org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:986) > > > > > > > > > > > > > > > at > > > > > > > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitializati > on(HMaster.java:780) > > > > > > > > > > > > > > > at > > > > > org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:183) > > > > > > > > > > > > > > > at > > org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1652) > > > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > > > > 2017-01-17 11:50:24,315 FATAL [bjsh19-16-30:16000.activeMasterManager] > > > > > master.HMaster: Master server abort: loaded coprocessors are: [] > > > > > > > > > > > > > > > 2017-01-17 11:50:24,316 FATAL [bjsh19-16-30:16000.activeMasterManager] > > > > > master.HMaster: Unhandled exception. Starting shutdown. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-01-17 11:27:17,926 INFO [regionserver/ > > > > > bjsh19-16-34.qbos.com/10.19.16.34:16020] regionserver.HRegionServer: > > > > > Serving as wjsa-tsl05,16020,1 > > > > > > > > > > > > > > > 484623636195, RpcServer on bjsh19-16-34.qbos.com/10.19.16.34:16020, > > > > > sessionid=0x154563e43e30179 > > > > > > > > > > > > > > > 2017-01-17 11:27:17,934 INFO [regionserver/ > > > > > bjsh19-16-34.qbos.com/10.19.16.34:16020] > > quotas.RegionServerQuotaManager: > > > > > Quota support disabled > > > > > > > > > > > > > > > 2017-01-17 11:27:23,966 INFO > > > > > [PriorityRpcServer.handler=14,queue=0,port=16020] > > > > > regionserver.RSRpcServices: Open hbase:namespace,,148462364327 > > > > > > > > > > > > > > > 9.30fab746cb3b6ceadcbda421459204b9. > > > > > > > > > > > > > > > 2017-01-17 11:27:24,008 WARN [RS_OPEN_REGION-bjsh19-16-34:16020-0] > > > > > zookeeper.ZKAssign: regionserver:16020-0x154563e43e30179, > > quorum=bjsh19-16 > > > > > > > > > > > > > > > -30:2181,bjsh19-16-31:2181,bjsh19-16-32:2181, baseZNode=/hbase115new > > > > > Attempt to transition the unassigned node for > 30fab746cb3b6ceadcbda421459 > > > > > > > > > > > > > > > 204b9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the > server > > > > > that tried to transition was wjsa-tsl05,16020,1484623636195 not the > > > > > > > > > > > > > > > expected bjsh19-16-34.qbos.com,16020,1484623636195 > > > > > > > > > > > > > > > 2017-01-17 11:27:24,008 WARN [RS_OPEN_REGION-bjsh19-16-34:16020-0] > > > > > coordination.ZkOpenRegionCoordination: Failed transition from OFFLINE > to > > O > > > > > > > > > > > > > > > PENING for region=30fab746cb3b6ceadcbda421459204b9 > > > > > > > > > > > > > > > 2017-01-17 11:27:24,008 WARN [RS_OPEN_REGION-bjsh19-16-34:16020-0] > > > > > handler.OpenRegionHandler: Region was hijacked? Opening cancelled for > > enco > > > > > > > > > > > > > > > dedName=30fab746cb3b6ceadcbda421459204b9 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
