[
https://issues.apache.org/jira/browse/HBASE-21192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612821#comment-16612821
]
stack edited comment on HBASE-21192 at 9/12/18 10:55 PM:
---------------------------------------------------------
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
h3. Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
h3. Why you need to cleanup STUCK regions.
Will cause buildup of master proc WALs IF an associated, outstanding procedure
(not always the case). If too many WALs, Master will be slow to start as it
runs through all outstanding WALs.
was (Author: stack):
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
h3. Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
h3. Why you need to cleanup STUCK regions.
Will cause buildup of master proc WALs. If too many, Master will be slow to
start as it runs through all outstanding WALs.
> Add HOW-TO repair damaged AMv2.
> -------------------------------
>
> Key: HBASE-21192
> URL: https://issues.apache.org/jira/browse/HBASE-21192
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
> Reporter: stack
> Assignee: stack
> Priority: Major
>
> Need a page or two on how to do various fixups. Will include doc on how to
> identify particular circumstance, how to run a repair, as well as caveats
> (e.g. if no log recovery, then region may be missing edits).
> Add pointer to log messages, especially those that explicitly ask for
> operator intervention; e.g. Master#inMeta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
