> The important bit will be that you get a currently working HBCK2 instance
I did. I downloaded all 2.2.0 artifacts locally, and build with the
parameters you gave. All went well. However, HBCK2 can't talk to my HBase
server...
I tried this:
java org.apache.hbase.HBCK2 assigns 1431292037213
But I got this:
PleaseHoldException: Master is initializing
The hbase hbck -j ~/hbase-hbck2-for-2.2.0.jar unassigns 1588230740 command
doesn.t seems to be correct. "-j" is not a valid parameter, should be -jar.
And fixing it displayes the previous HBCK version help.
Because the META is now assigned, I changed 1588230740 to 1431292037213 to
get the namespace table assigned. No idea how the meta got assigned... I
removed /hbase from ZK, maybe it helped.
Cluster is still running in maintenance mode:
<property>
<name>hbase.master.maintenance_mode</name>
<value>true</value>
</property>
That's why I'm very surprised it tries to still assign some regions to
other servers (master is node2)
2019-03-12 10:59:12,286 WARN [master/node2:60000:becomeActiveMaster]
master.HMaster:
hbase:namespace,,1431292037213.7f4a480f47f98300185d1ae2ff663295. is NOT
online; state={7f4a480f47f98300185d1ae2ff663295 state=OPENING,
ts=1552402749342, server=node3,16020,1551984565705};
ServerCrashProcedures=true. Master startup cannot progress, in
holding-pattern until region onlined.
I looked at node3 and there is absolutely nothing wrong there. Logs are
just showing LruBlockCacheStatsExecutor. There is nothing there but INFO
logs.
Thanks again for all your suggestions!
JMS
Le mar. 12 mars 2019 à 13:23, Wellington Chevreuil <
[email protected]> a écrit :
> >
> > server=node3,60000,-1
>
>
> That's an unusual server start code (-1). Maybe that's what's causing meta
> to never come online, as there will be no SCP for this RS instance. The
> hbck2 unassigns command Busbey suggested will probably clean assignment for
> meta, than you could try the "assigns" command to bring it online again.
>
> Em ter, 12 de mar de 2019 às 15:30, Sean Busbey <[email protected]>
> escreveu:
>
> > On Tue, Mar 12, 2019 at 9:38 AM Jean-Marc Spaggiari
> > <[email protected]> wrote:
> > >
> > > Ok. First, thanks to all of you for taking a look at that!
> > >
> > > Sean, I didn't follow the steps. 2.0.4 was failure and not able to
> > start. I
> > > got the recommendation to remove the master wall, which I did. So I
> did't
> > > set the procedure upgrade tag, because there was no more any data on
> the
> > > procedure side. Should I still put it even if I wiped everything?
> > >
> >
> > Probably not? Let's see if we can recover. :)
> >
> > I've already pushed to get the "here is how you upgrade" doc made more
> > visible, so hopefully others don't end up here.
> >
> >
> >
> > >
> > > "Can you tell me where in this high level things fell down? Or where I
> > > should drill in more?"
> > >
> > > It's hard to say. I tried way too many things I think, to be able to
> > point
> > > to something specific. I think HBCK2 is definitely somewhere where I
> > > struggled. So the instructions you sent are very helpful.
> > >
> >
> >
> > The important bit will be that you get a currently working HBCK2
> instance.
> >
> >
> > >
> > > 2019-03-12 10:37:08,385 WARN [master/node2:60000:becomeActiveMaster]
> > > master.HMaster: hbase:meta,,1.1588230740 is NOT online;
> state={1588230740
> > > state=OPEN, ts=1552400910746, server=node3,60000,-1};
> > > ServerCrashProcedures=true. Master startup cannot progress, in
> > > holding-pattern until region onlined.
> > >
> >
> > excellent! this is great info. This says that the master has no RS
> > report that shows meta as happy, but also that internally the master
> > thinks meta is open on RS "node3" so it isn't trying to recover it.
> >
> > Is node3 up? If it is, does the web UI for node3 show meta as a table
> > it's serving? does the RS log for node3 say anything about meta?
> >
> > If node3 isn't currently struggling with some issue, let's assume you
> > are just in a bad state from when you were running 2.0. In that case
> > we can use hbck2 and tell master to unassign meta:
> >
> > $ hbase hbck -j ~/hbase-hbck2-for-2.2.0.jar unassigns 1588230740
> >
>
