Master log contained entries in the following form: 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK exec-time=242hrs, 10mins, 28.896sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
Note the procedure Id was around 43000, far lower than 147464. Can you turn debug log on and repost master log ? Thanks On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <adam.cec...@hitec.lu> wrote: > Hello, > > > We're having a table stuck in disabled state. First I'd like to start with > what I tried already: > > > * Restart all machines involved in HBase cluster > > * hbase hbck with varios arguments > > * hdfs fsck > > * Purge ZK /hbase and restart masters > > > Now more details anout what happens: > > * When enabling from hbase shell: > > > hbase(main):002:0> enable "sentinel-meta" > ERROR: The procedure 147464 is still running > > > The task ID changes every time I run the command so I think it's talking > about itself (and it gets stuck for a while before saying anything) > > > In the log, all I can see is: > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK > exec-time=242hrs, 52mins, 7.454sec > exception=org.apache.hadoop.hbase.TableNotEnabledException: > sentinel-meta > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] > procedure.DisableTableProcedure: > Table sentinel-meta isn't enabled; skipping disable > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK > exec-time=242hrs, 51mins, 42.288sec > exception=org.apache.hadoop.hbase.TableNotEnabledException: > sentinel-meta > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] > procedure.DisableTableProcedure: > Table sentinel-meta isn't enabled; skipping disable > > > Please also find attached a complete log from startup to shutdown on a > single active master. You'll see the table is found as well as the regions > but it gets deactivated with no reason. > > > Thanks a lot for your help, we're kinda running out if ideas here. > > > Best regards, > > > Adam. > > >
