Josh Elser created HBASE-23011:
----------------------------------
Summary: AP stuck in retry loop if underlying table no longer
exists
Key: HBASE-23011
URL: https://issues.apache.org/jira/browse/HBASE-23011
Project: HBase
Issue Type: Bug
Affects Versions: 2.1.6, 2.0.6
Reporter: Josh Elser
Assignee: Josh Elser
Looking at a user's issue with [~wchevreuil]... While the details of how
exactly we got into this situation are murky, I'm noticing that we have a
situation where an AP can get stuck resubmitting itself over and over if,
somehow, the table the region the AP is assigning gets deleted.
{noformat}
2019-08-25 23:33:54,588 WARN [PEWorker-11]
assignment.RegionTransitionProcedure: Failed transition, suspend 1secs
pid=1100250, ppid=1100195, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true;
AssignProcedure table=<tablename>, region=<regionid>; rit=OFFLINE,
location=null; waiting on rectified condition fixed by other Procedure or
operator intervention
org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException:
monitoring:test1
at
org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:215)
at
org.apache.hadoop.hbase.master.assignment.AssignProcedure.assign(AssignProcedure.java:195)
at
org.apache.hadoop.hbase.master.assignment.AssignProcedure.startTransition(AssignProcedure.java:206)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:364)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:98)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:958)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1836)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1596)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1200(ProcedureExecutor.java:80)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2141)
{noformat}
Stack trace looks like similar to the above.
The problem appears to be that we don't catch the
{{TableStateNotFoundException}} coming out of
{{TableStateManager#getTableState(TableName)}}. This keeps the AP in a
fail/resubmit loop (until, presumably, someone comes along with an `HBCK2
bypass`). This is only a problem in branch-2.0 and branch-2.1.
{{TransitRegionStateProcedure}} in branch-2.2+ doesn't have the same issue (at
least on the surface).
As mentioned earlier, it's not clear how we got this SCP(1100195)->AP(1100250)
scheduled while the table itself is actually deleted. Some quick attempts to
reproduce this locally weren't successful. I'm not sure if I can write a
meaningful test. Need to try to look more closely at that, but will attach a
patch which I think will work around the issue.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
