[ 
https://issues.apache.org/jira/browse/HBASE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433324#comment-16433324
 ] 

Xiaolin Ha commented on HBASE-20368:
------------------------------------

[~stack]The case is that all the regionservers are stopped(It's better to wait 
for a while), and then we start one or more of them or add new servers to this 
rsgroup. 
There are some differences between this case and restarting all the regions 
servers in the rsgroup.
 '...regions on this rsgroup will be reassigned, but there is no available 
servers of this rsgroup....' It means when all the regionservers in the rsgroup 
are offline, the assginment of regions will be failed. But the problem is that 
when some servers in the rsgroup are online again, the assignment of the 
regions will not be continued because AM's pendingAssginQueue was cleared after 
the last assginment though no available servers were found.

We can see the stuck of RIT by logs or UI, and DML of tables in this rsgroup 
will also show it.
Logs are: 
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st23.bj,40100,1523172960034, table=t3, 
region=c8890704468083ceae6a6c3b5e24b968
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st26.bj,40100,1523172965147, table=hh:t3, 
region=97591999e282ac4dc54300693bba4263
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st23.bj,40100,1523172960034, table=t1, 
region=62bdc6fb8e9af1c21a323c9191313613
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st23.bj,40100,1523172960034, table=t2, 
region=9bdf2635e2d76c0d0388a5708ce21e3c
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st26.bj,40100,1523172965147, table=hh:t1, 
region=47ba2a3d6968ad09a79e05bdd6db5694
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st23.bj,40100,1523172960034, table=t2, 
region=ddef57619b45a023e076c3d5bcf30a04
2018-04-09,11:48:39,421 WARN 
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK 
Region-In-Transition rit=OFFLINE, 
location=c3-hadoop-tst-st26.bj,40100,1523172965147, table=t2, 
region=c0fb941b4c27fa04211e119494cf34d1






> Fix RIT stuck when a rsgroup has no online servers but AM's 
> pendingAssginQueue is cleared
> -----------------------------------------------------------------------------------------
>
>                 Key: HBASE-20368
>                 URL: https://issues.apache.org/jira/browse/HBASE-20368
>             Project: HBase
>          Issue Type: Bug
>          Components: rsgroup
>    Affects Versions: 2.0.0
>            Reporter: Xiaolin Ha
>            Assignee: Xiaolin Ha
>            Priority: Major
>         Attachments: HBASE-20368.branch-2.0.001.patch
>
>
> This error can be reproduced by shutting down all servers in a rsgroups and 
> starting them soon afterwards. 
> The regions on this rsgroup will be reassigned, but there is no available 
> servers of this rsgroup.
> They will be added to AM's pendingAssginQueue, which AM will clear regardless 
> of the result of assigning in this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to