[
https://issues.apache.org/jira/browse/HBASE-30323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Kumar Kumawat updated HBASE-30323:
----------------------------------------
Description:
region_mover.rb should only choose same rsgroup servers as target servers. In
current case HMaster need to correct it. I think we can skip the process where
HMaster tries to figure out the target host if we pass the right one.
As HMaster do have one layer of protection, on its own it doesn't post any
issue but if this issue comes with other issue like "drainig host is not added
to draining list", then it can cause unavailibility.
We faced it in out prod where we use different RS group for system tables ie
hbase:meta.
Lets say hbase:meta is on rs-sys-1.
Before start of draining the operation where we add it to draining list failed.
Draining script tried to drain the rs-sys-1. it tried to assign the hbase:meta
on rs-x. HMaster prevented it but it choose rs-sys-0 only that was about to be
killed.
This caused hbase:meta to unavailable for some time. And rolling restart at the
same time can cause more unavailability of other data regions.
draining_servers.rb add command timed out so HMaster was not aware of it. (We
can have some improvements here as well. Going ahead with drainig when HMaster
is not aware is also not right)
{{}}
{code:java}
2026-06-03 14:07:29,341 INFO [PEWorker-39]
assignment.TransitRegionStateProcedure - Starting pid=193574338,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true;
TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE;
state=CLOSED, location=localhost,1,1; forceNewPlan=false, retain=false
2026-06-03 14:07:28,550 INFO [PEWorker-54] procedure2.ProcedureExecutor -
Initialized subprocedures=[{pid=193574380, ppid=193574338, state=RUNNABLE;
CloseRegionProcedure 1588230740, server=regionserver-sys-1}]
2026-06-03 14:07:29,504 INFO [PEWorker-27] procedure2.ProcedureExecutor -
Initialized subprocedures=[{pid=193574527, ppid=193574338, state=RUNNABLE;
OpenRegionProcedure 1588230740, server=regionserver-sys-1}]
{code}
{{}}
{{As this RS was already scheduled to be killed. It got killed after some time.
}}
{{}}
{code:java}
2026-06-03 14:10:39,022 INFO [PEWorker-18] procedure.MasterProcedureScheduler -
Took xlock for pid=193622058, ppid=193622049,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN{code}
{{{}{}}}{{{}{}}}{{{}{}}}{{{}{}}}
was:
region_mover.rb should only choose same rsgroup servers as target servers. In
current case HMaster need to correct it. I think we can skip the process where
HMaster tries to figure out the target host if we pass the right one.
> [RSGroup] Forward-port HBASE-22658 to branch-2
> ----------------------------------------------
>
> Key: HBASE-30323
> URL: https://issues.apache.org/jira/browse/HBASE-30323
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
> Affects Versions: 2.0.6, 2.2.3, 2.1.9
> Reporter: Umesh Kumar Kumawat
> Assignee: Umesh Kumar Kumawat
> Priority: Major
> Fix For: 3.0.0-alpha-1, 3.0.0
>
>
> region_mover.rb should only choose same rsgroup servers as target servers.
> In current case HMaster need to correct it. I think we can skip the process
> where HMaster tries to figure out the target host if we pass the right one.
>
> As HMaster do have one layer of protection, on its own it doesn't post any
> issue but if this issue comes with other issue like "drainig host is not
> added to draining list", then it can cause unavailibility.
> We faced it in out prod where we use different RS group for system tables ie
> hbase:meta.
> Lets say hbase:meta is on rs-sys-1.
> Before start of draining the operation where we add it to draining list
> failed. Draining script tried to drain the rs-sys-1. it tried to assign the
> hbase:meta on rs-x. HMaster prevented it but it choose rs-sys-0 only that was
> about to be killed.
> This caused hbase:meta to unavailable for some time. And rolling restart at
> the same time can cause more unavailability of other data regions.
>
> draining_servers.rb add command timed out so HMaster was not aware of it.
> (We can have some improvements here as well. Going ahead with drainig when
> HMaster is not aware is also not right)
> {{}}
> {code:java}
> 2026-06-03 14:07:29,341 INFO [PEWorker-39]
> assignment.TransitRegionStateProcedure - Starting pid=193574338,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE;
> state=CLOSED, location=localhost,1,1; forceNewPlan=false, retain=false
>
> 2026-06-03 14:07:28,550 INFO [PEWorker-54] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=193574380, ppid=193574338, state=RUNNABLE;
> CloseRegionProcedure 1588230740, server=regionserver-sys-1}]
>
> 2026-06-03 14:07:29,504 INFO [PEWorker-27] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=193574527, ppid=193574338, state=RUNNABLE;
> OpenRegionProcedure 1588230740, server=regionserver-sys-1}]
> {code}
> {{}}
> {{As this RS was already scheduled to be killed. It got killed after some
> time. }}
> {{}}
> {code:java}
> 2026-06-03 14:10:39,022 INFO [PEWorker-18] procedure.MasterProcedureScheduler
> - Took xlock for pid=193622058, ppid=193622049,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN{code}
> {{{}{}}}{{{}{}}}{{{}{}}}{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
