I removed the node.dc1.consul from resolve.conf and restarted the cluster but it still shows up on the master UI.
amb2.node.dc1.consul,16020,1448353564099Tue Nov 24 08:26:04 UTC 201500 amb2.service.consul,16020,1448353564099Tue Nov 24 08:26:04 UTC 201500 The logs report [1] that the meta region fails to assign to node.dc1.consul and then tries to assign it to amb2.service.consul and gets stuck in PENDING_OPEN again. --- 1588230740hbase:meta,,1.1588230740 state=PENDING_OPEN, ts=Tue Nov 24 09:32:26 UTC 2015 (450s ago), server=amb2.service.consul,16020,1448357534179450511 --- Before I restarted the cluster, the master log [2] complained about not being able to connect to amb2.node.dc1.consul/172.17.0.85:16020. Im not sure but somehow it feels as if amb2.node.dc1.consul shadows the real host amb2.service.consul. I was looking into the source code and found the configuration 'hbase.regionserver.hostname' - could that be of help here to remove the node.dc1 host? [1] http://pastebin.com/uZKqK9BJ [2] http://pastebin.com/s10E2rtA On Tue, Nov 24, 2015 at 10:23 AM, Samir Ahmic <[email protected]> wrote: > Hi Kristoffer, > It looks like you have some issue with name resolution. Try to remove > incorrect value from reslove.conf (node.dc1.consul) and then restart hbase > cluster. > Regarding issue with region in transition check master log for > "hbase:meta,,1.1588230740" > there should be exception explaining why hbase:meta can to be transition > from PENDING_OPEN to OPEN state, if hbase:meta table is unavailable master > can not finish initialization. > > Regards > Samir > > On Tue, Nov 24, 2015 at 10:11 AM, Kristoffer Sjögren <[email protected]> > wrote: > >> Sorry, I should mention that this is HBase 1.1.2. >> >> Zookeeper only report one region server. >> >> $ ls /hbase-unsecure/rs >> [amb2.service.consul,16020,1448353564099] >> >> >> >> >> On Tue, Nov 24, 2015 at 9:55 AM, Kristoffer Sjögren <[email protected]> >> wrote: >> > Hi >> > >> > I'm trying to install a HBase cluster with 1 master >> > (amb1.service.consul) and 1 region server (amb2.service.consul) using >> > Ambari on docker containers provided by sequenceiq [1] using a custom >> > blueprint [2]. >> > >> > Every component installs correctly except for HBase which get stuck >> > with regions in transition: >> > >> > --- >> > hbase:meta,,1.1588230740 state=PENDING_OPEN, ts=Tue Nov 24 08:26:45 >> > UTC 2015 (1098s ago), server=amb2.service.consul,16020,1448353564099 >> > --- >> > >> > And for some reason 2 region servers (instead of 1) are discovered by >> > the master with the exact same timestamp but with different hostnames. >> > I'm not sure if this is the reason why the regions get stuck. >> > >> > ---- >> > amb2.node.dc1.consul,16020,1448353564099Tue Nov 24 08:26:04 UTC 201500 >> > amb2.service.consul,16020,1448353564099Tue Nov 24 08:26:04 UTC 201500 >> > ---- >> > >> > The only place I can find "amb2.node.dc1.consul" on the ambari >> > agent/server hosts is in /etc/resolv.conf which looks like this. >> > >> > ---- >> > nameserver 172.17.0.82 >> > search service.consul node.dc1.consul >> > ---- >> > >> > Is there some way that I can manually tell the master to disregard the >> > "phantom" host amb2.node.dc1.consul? >> > >> > Any help or tips appreciated. >> > >> > Cheers, >> > -Kristoffer >> > >> > >> > [1] https://github.com/sequenceiq/docker-ambari >> > [2] >> https://gist.githubusercontent.com/krisskross/901ed8223c1ed1db80e3/raw/869327be9ad15e6a9f099a7591323244cd245357/ambari-hdp2.3 >>
