RE: Table is disabled an no way to get it back online
Hello, Thanks a lot, the table is back online. One last question ? Can you provide a log pattern to spot this ? Just in case it occurs again ;-) Regards, Adam. De : Matteo Bertozzi <theo.berto...@gmail.com> Envoyé : mardi 22 novembre 2016 19:48 À : user@hbase.apache.org Objet : Re: Table is disabled an no way to get it back online hadoop fs -rmr /hbase/MasterProcWALs Matteo On Tue, Nov 22, 2016 at 10:42 AM, Cecile, Adam <adam.cec...@hitec.lu> wrote: > Can you explain me how to delete masterwals directory? > > > > Sent from my Samsung device > > > Original message > From: Matteo Bertozzi <theo.berto...@gmail.com> > Date: 22/11/2016 19:12 (GMT+01:00) > To: user@hbase.apache.org > Subject: Re: Table is disabled an no way to get it back online > > I don't think this has anything to do with HBASE-13415 or the bugfix it > related to it i'm working on. > > this is probably the usual case of mismatch state with zk. Disable is > saying that the table is already not disabled. > so, enableTable() is the one that should give any exception in case. but I > don't see any. > I suggest to just drop the MasterWALs directory, drop the znode > /hbase/table/sentinel-meta, restart the master and try disable and then > enable. that should bring you back with the table online > > Matteo > > > On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <adam.cec...@hitec.lu> > wrote: > > > Thanks for everything. As you said, this bug is supposed to be fixed in > > 1.2.0. Matteo is reading this list as well ? > > > > Regards, Adam. > > > > De : Ted Yu <yuzhih...@gmail.com> > > Envoyé : mardi 22 novembre 2016 19:00 > > À : user@hbase.apache.org > > Objet : Re: Table is disabled an no way to get it back online > > > > Please take a look at HBASE-13415 > > > > From the log, you're using hbase 1.2.0 already. But I heard there is a > > subtle bug which is being fixed. > > > > Matteo is the person with best knowledge in this regard. > > > > On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <adam.cec...@hitec.lu> > > wrote: > > > > > Another one, because I'm not sure the log is overwritten when > restarting. > > > This one has been cleared before service start. > > > > > > De : Cecile, Adam <adam.cec...@hitec.lu> > > > Envoyé : mardi 22 novembre 2016 18:42 > > > À : user@hbase.apache.org > > > Objet : RE: Table is disabled an no way to get it back online > > > > > > Hello, > > > > > > Sadly I could not use the webui, it killed my firefox (probably way too > > > much time). Here is the debug log... (11Mb uncompressed for maybe two > > > minutes running !!) > > > > > > Best regards, Adam. > > > > > > De : Ted Yu <yuzhih...@gmail.com> > > > Envoyé : mardi 22 novembre 2016 17:05 > > > À : user@hbase.apache.org > > > Objet : Re: Table is disabled an no way to get it back online > > > > > > In log4j.properties : > > > > > > log4j.logger.org.apache.hadoop.hbase=DEBUG > > > > > > On master UI, you can select the Procedures tab. Pastebin what you see > > > (text is enough). > > > > > > Thanks > > > > > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <adam.cec...@hitec.lu> > > > wrote: > > > > > > > Hey Ted, > > > > > > > > Thank you. Heading home right now but I'll start the laptop again. > Not > > > > sure exactly how I should turn debug log on so if you have the > > > information > > > > it'd be appreciated, otherwise I'll look at the xml files. > > > > > > > > Regards, Adam. > > > > > > > > De : Ted Yu <yuzhih...@gmail.com> > > > > Envoyé : mardi 22 novembre 2016 15:46 > > > > À : user@hbase.apache.org > > > > Objet : Re: Table is disabled an no way to get it back online > > > > > > > > Master log contained entries in the following form: > > > > > > > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] > > > > procedure2.ProcedureExecutor: Rolledback procedure > > DisableTableProcedure > > > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK > > > > exec-time=242hrs, 10mins, 28.896sec &
RE: Table is disabled an no way to get it back online
Can you explain me how to delete masterwals directory? Sent from my Samsung device Original message From: Matteo Bertozzi <theo.berto...@gmail.com> Date: 22/11/2016 19:12 (GMT+01:00) To: user@hbase.apache.org Subject: Re: Table is disabled an no way to get it back online I don't think this has anything to do with HBASE-13415 or the bugfix it related to it i'm working on. this is probably the usual case of mismatch state with zk. Disable is saying that the table is already not disabled. so, enableTable() is the one that should give any exception in case. but I don't see any. I suggest to just drop the MasterWALs directory, drop the znode /hbase/table/sentinel-meta, restart the master and try disable and then enable. that should bring you back with the table online Matteo On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <adam.cec...@hitec.lu> wrote: > Thanks for everything. As you said, this bug is supposed to be fixed in > 1.2.0. Matteo is reading this list as well ? > > Regards, Adam. > > De : Ted Yu <yuzhih...@gmail.com> > Envoyé : mardi 22 novembre 2016 19:00 > À : user@hbase.apache.org > Objet : Re: Table is disabled an no way to get it back online > > Please take a look at HBASE-13415 > > From the log, you're using hbase 1.2.0 already. But I heard there is a > subtle bug which is being fixed. > > Matteo is the person with best knowledge in this regard. > > On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <adam.cec...@hitec.lu> > wrote: > > > Another one, because I'm not sure the log is overwritten when restarting. > > This one has been cleared before service start. > > > > De : Cecile, Adam <adam.cec...@hitec.lu> > > Envoyé : mardi 22 novembre 2016 18:42 > > À : user@hbase.apache.org > > Objet : RE: Table is disabled an no way to get it back online > > > > Hello, > > > > Sadly I could not use the webui, it killed my firefox (probably way too > > much time). Here is the debug log... (11Mb uncompressed for maybe two > > minutes running !!) > > > > Best regards, Adam. > > > > De : Ted Yu <yuzhih...@gmail.com> > > Envoyé : mardi 22 novembre 2016 17:05 > > À : user@hbase.apache.org > > Objet : Re: Table is disabled an no way to get it back online > > > > In log4j.properties : > > > > log4j.logger.org.apache.hadoop.hbase=DEBUG > > > > On master UI, you can select the Procedures tab. Pastebin what you see > > (text is enough). > > > > Thanks > > > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <adam.cec...@hitec.lu> > > wrote: > > > > > Hey Ted, > > > > > > Thank you. Heading home right now but I'll start the laptop again. Not > > > sure exactly how I should turn debug log on so if you have the > > information > > > it'd be appreciated, otherwise I'll look at the xml files. > > > > > > Regards, Adam. > > > > > > De : Ted Yu <yuzhih...@gmail.com> > > > Envoyé : mardi 22 novembre 2016 15:46 > > > À : user@hbase.apache.org > > > Objet : Re: Table is disabled an no way to get it back online > > > > > > Master log contained entries in the following form: > > > > > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] > > > procedure2.ProcedureExecutor: Rolledback procedure > DisableTableProcedure > > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK > > > exec-time=242hrs, 10mins, 28.896sec > > > exception=org.apache.hadoop.hbase.TableNotEnabledException: > > sentinel-meta > > > > > > Note the procedure Id was around 43000, far lower than 147464. > > > > > > Can you turn debug log on and repost master log ? > > > > > > Thanks > > > > > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <adam.cec...@hitec.lu> > > > wrote: > > > > > > > Hello, > > > > > > > > > > > > We're having a table stuck in disabled state. First I'd like to start > > > with > > > > what I tried already: > > > > > > > > > > > > * Restart all machines involved in HBase cluster > > > > > > > > * hbase hbck with varios arguments > > > > > > > > * hdfs fsck > > > > > > > > * Purge ZK /hbase and restart masters > > > > > > > > > > > > Now more detai
RE: Table is disabled an no way to get it back online
Thanks for everything. As you said, this bug is supposed to be fixed in 1.2.0. Matteo is reading this list as well ? Regards, Adam. De : Ted Yu <yuzhih...@gmail.com> Envoyé : mardi 22 novembre 2016 19:00 À : user@hbase.apache.org Objet : Re: Table is disabled an no way to get it back online Please take a look at HBASE-13415 >From the log, you're using hbase 1.2.0 already. But I heard there is a subtle bug which is being fixed. Matteo is the person with best knowledge in this regard. On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <adam.cec...@hitec.lu> wrote: > Another one, because I'm not sure the log is overwritten when restarting. > This one has been cleared before service start. > ____ > De : Cecile, Adam <adam.cec...@hitec.lu> > Envoyé : mardi 22 novembre 2016 18:42 > À : user@hbase.apache.org > Objet : RE: Table is disabled an no way to get it back online > > Hello, > > Sadly I could not use the webui, it killed my firefox (probably way too > much time). Here is the debug log... (11Mb uncompressed for maybe two > minutes running !!) > > Best regards, Adam. > > De : Ted Yu <yuzhih...@gmail.com> > Envoyé : mardi 22 novembre 2016 17:05 > À : user@hbase.apache.org > Objet : Re: Table is disabled an no way to get it back online > > In log4j.properties : > > log4j.logger.org.apache.hadoop.hbase=DEBUG > > On master UI, you can select the Procedures tab. Pastebin what you see > (text is enough). > > Thanks > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <adam.cec...@hitec.lu> > wrote: > > > Hey Ted, > > > > Thank you. Heading home right now but I'll start the laptop again. Not > > sure exactly how I should turn debug log on so if you have the > information > > it'd be appreciated, otherwise I'll look at the xml files. > > > > Regards, Adam. > > > > De : Ted Yu <yuzhih...@gmail.com> > > Envoyé : mardi 22 novembre 2016 15:46 > > À : user@hbase.apache.org > > Objet : Re: Table is disabled an no way to get it back online > > > > Master log contained entries in the following form: > > > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] > > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK > > exec-time=242hrs, 10mins, 28.896sec > > exception=org.apache.hadoop.hbase.TableNotEnabledException: > sentinel-meta > > > > Note the procedure Id was around 43000, far lower than 147464. > > > > Can you turn debug log on and repost master log ? > > > > Thanks > > > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <adam.cec...@hitec.lu> > > wrote: > > > > > Hello, > > > > > > > > > We're having a table stuck in disabled state. First I'd like to start > > with > > > what I tried already: > > > > > > > > > * Restart all machines involved in HBase cluster > > > > > > * hbase hbck with varios arguments > > > > > > * hdfs fsck > > > > > > * Purge ZK /hbase and restart masters > > > > > > > > > Now more details anout what happens: > > > > > > * When enabling from hbase shell: > > > > > > > > > hbase(main):002:0> enable "sentinel-meta" > > > ERROR: The procedure 147464 is still running > > > > > > > > > The task ID changes every time I run the command so I think it's > talking > > > about itself (and it gets stuck for a while before saying anything) > > > > > > > > > In the log, all I can see is: > > > > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] > > > procedure2.ProcedureExecutor: Rolledback procedure > DisableTableProcedure > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase. > > TableNotEnabledException: > > > sentinel-meta > > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure. > > DisableTableProcedure: > > > Table sentinel-meta isn't enabled; skipping disable > > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] > > > procedure2.ProcedureExecutor: Rolledback procedure > DisableTableProcedure > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase. > > TableNotEnabledException: > > > sentinel-meta > > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure. > > DisableTableProcedure: > > > Table sentinel-meta isn't enabled; skipping disable > > > > > > > > > Please also find attached a complete log from startup to shutdown on a > > > single active master. You'll see the table is found as well as the > > regions > > > but it gets deactivated with no reason. > > > > > > > > > Thanks a lot for your help, we're kinda running out if ideas here. > > > > > > > > > Best regards, > > > > > > > > > Adam. > > > > > > > > > > > >
RE: Table is disabled an no way to get it back online
Hey Ted, Thank you. Heading home right now but I'll start the laptop again. Not sure exactly how I should turn debug log on so if you have the information it'd be appreciated, otherwise I'll look at the xml files. Regards, Adam. De : Ted Yu <yuzhih...@gmail.com> Envoyé : mardi 22 novembre 2016 15:46 À : user@hbase.apache.org Objet : Re: Table is disabled an no way to get it back online Master log contained entries in the following form: 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK exec-time=242hrs, 10mins, 28.896sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta Note the procedure Id was around 43000, far lower than 147464. Can you turn debug log on and repost master log ? Thanks On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <adam.cec...@hitec.lu> wrote: > Hello, > > > We're having a table stuck in disabled state. First I'd like to start with > what I tried already: > > > * Restart all machines involved in HBase cluster > > * hbase hbck with varios arguments > > * hdfs fsck > > * Purge ZK /hbase and restart masters > > > Now more details anout what happens: > > * When enabling from hbase shell: > > > hbase(main):002:0> enable "sentinel-meta" > ERROR: The procedure 147464 is still running > > > The task ID changes every time I run the command so I think it's talking > about itself (and it gets stuck for a while before saying anything) > > > In the log, all I can see is: > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK > exec-time=242hrs, 52mins, 7.454sec > exception=org.apache.hadoop.hbase.TableNotEnabledException: > sentinel-meta > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] > procedure.DisableTableProcedure: > Table sentinel-meta isn't enabled; skipping disable > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK > exec-time=242hrs, 51mins, 42.288sec > exception=org.apache.hadoop.hbase.TableNotEnabledException: > sentinel-meta > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] > procedure.DisableTableProcedure: > Table sentinel-meta isn't enabled; skipping disable > > > Please also find attached a complete log from startup to shutdown on a > single active master. You'll see the table is found as well as the regions > but it gets deactivated with no reason. > > > Thanks a lot for your help, we're kinda running out if ideas here. > > > Best regards, > > > Adam. > > >
