Hi hbase comunity.
In our production environment we have hbase 2.2.6 with hadoop 2.8.5.We found
that when we do a simple reboot of an hbase regionserverafter a configuration
change or maintenance, very often regions go into transition.
the problem is that some regions often get stuck during the transition and
never recover on their own.
On the master's logs we usually see:2021-12-20 12:21:17,004 WARN
[ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition
rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_7034_hnotifevents, region=3739acab527c013a04249e5bf83756392021-12-20
12:21:17,004 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_618_hnotifevents, region=9d5f39b2022d42127366a6d98f4dc67b2021-12-20
12:21:17,004 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_7854_huserlog, region=52be50fb036cde773fc5a13bd2d8547d2021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_2469_huserlog, region=b6e38248cc3336349612e50e790b9cc52021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10,16020,1639998430080,
table=mn1_4871_hinvio, region=c8570afd26643e03879708df2444ea6d2021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_3856_huserlog, region=f6a21f35f1265f9bed95cb99c8f008602021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_4871_hinvio, region=2a91d2da06b3b0082d42825a140fc4982021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_618_hwfsessionh, region=950e0cbd96b7f1691eda531ddd446dd82021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_3856_hevents, region=2dec9ce47bccb2daa92da120e56d9af42021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_4639_hevents, region=65bdc201b104059d4d55f9ebe955e7bc2021-12-20
12:21:17,005 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING, location=rzv-db10-hd,16020,1639998430080,
table=mn1_942_huserlog, region=cb3f422f56ccc29d30f3506ea7041425
We always fixed it by restarting the location in STUCK.
- Is there the possibility of avoiding the transition of regions when doing a
simple reboot of a node for configuration change?- Is it normal for transitions
to time out and not recover on their own? Is there any way to avoid this
problem?
Thank you,
Hamado Dene
