[ 
https://issues.apache.org/jira/browse/HBASE-28690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Umesh Kumar Kumawat updated HBASE-28690:
----------------------------------------
    Description: 
A {{CloseRegionProcedure on master request the RS to close the region and after 
closing the region RS reports RegionStateTransition 
back([here|https://github.com/apache/hbase/blob/d1015a68ed9f94d74668abd37edefd32f5e9305b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1853]).
 On recieving the report, master checks if }}regionNode has any procedure 
assigned to it 
([code|https://github.com/apache/hbase/blob/d1015a68ed9f94d74668abd37edefd32f5e9305b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1294]).
 

 
{code:java}
 private boolean reportTransition(RegionStateNode regionNode, ServerStateNode 
serverNode,
    TransitionCode state, long seqId, long procId) throws IOException {
    ServerName serverName = serverNode.getServerName();
    TransitRegionStateProcedure proc = regionNode.getProcedure();
    if (proc == null) {
      return false;
    }
    proc.reportTransition(master.getMasterProcedureExecutor().getEnvironment(), 
regionNode,
      serverName, state, seqId, procId);
    return true;
  } {code}
If regionNode doesn't have any procedure, the master just logs it and doesn't 
throw any error to RPC. 

 

Think of a case when MasterFailover is happening and the new Active master only 
initialized the TRSP and CloseRegionProcedure. Now aborting Master has 
stale/false data. If the transition report comes to the aborting master, not 
rejecting this report is causing the procedure to stuck. 

 

*Logs for more understanding* 

active master server4-1 failing

{{}}
{noformat}
2024-06-20 04:45:05,576 ERROR 
[iority.RWQ.Fifo.write.handler=3,queue=0,port=61000] master.HMaster - ***** 
ABORTING master server4-1,61000,1715413775736: Failed to record region server 
as started *****{noformat}
{{}}

 

 

*logs of new active master server5-1*

 
{noformat}
2024-06-20 04:49:28,893 DEBUG [aster/server5-1:61000:becomeActiveMaster] 
assignment.RegionStateStore - Load hbase:meta entry 
region=888a715d5926adbb89c985d8967f40d4, regionState=OPEN, 
lastHost=server1-119,61020,1717560166420, 
regionLocation=server1-119,61020,1717560166420, openSeqNum=34892620

024-06-20 04:49:51,886 INFO [PEWorker-22] procedure2.ProcedureExecutor - 
Initialized subprocedures=[{pid=16276416, ppid=16276108, 
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure 
table=RIMBS.UPLOADER_JOB_DETAILS, region=888a715d5926adbb89c985d8967f40d4, 
UNASSIGN}]  (on server5-1)

2024-06-20 04:49:52,022 INFO [PEWorker-40] procedure2.ProcedureExecutor - 
Initialized subprocedures=[{pid=16276470, ppid=16276416, state=RUNNABLE; 
CloseRegionProcedure 888a715d5926adbb89c985d8967f40d4, 
server=server1-119,61020,1717560166420}] (on server5-1){noformat}
 

 

*RS logs for closing* 
{noformat}
2024-06-20 04:49:52,267 INFO [_REGION-regionserver/server1-119:61020-2] 
handler.UnassignRegionHandler - Close 888a715d5926adbb89c985d8967f40d4

2024-06-20 04:49:52,267 DEBUG [_REGION-regionserver/server1-119:61020-2] 
regionserver.HRegion - Closing 888a715d5926adbb89c985d8967f40d4, disabling 
compactions & flushes

2024-06-20 04:49:52,354 INFO [_REGION-regionserver/server1-119:61020-2] 
regionserver.HRegion - Closed 
TABLE,KW\x00na240-app1-16\x00/Events-120620231740\x00MARKER-Events,1702619592612.888a715d5926adbb89c985d8967f40d4.

{noformat}
*Logs of report on aborting active Hmaster*
{noformat}
2024-06-20 04:49:52,355 WARN 
[iority.RWQ.Fifo.write.handler=1,queue=0,port=61000] 
assignment.AssignmentManager - No matching procedure found for 
server1-119,61020,1717560166420 transition on state=OPEN, 
location=server1-119,61020,1717560166420, table=RIMBS.UPLOADER_JOB_DETAILS, 
region=888a715d5926adbb89c985d8967f40d4 to CLOSED ( host = server4-1 , 
hbaseMasterLogFile){noformat}

  was:
A {{CloseRegionProcedure on master request the RS to close the region and after 
closing the region RS reports RegionStateTransition 
back([here|https://github.com/apache/hbase/blob/d1015a68ed9f94d74668abd37edefd32f5e9305b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1853]).
 On recieving the report, master checks if }}regionNode has any procedure 
assigned to it 
([code|https://github.com/apache/hbase/blob/d1015a68ed9f94d74668abd37edefd32f5e9305b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1294]).
 

 
{code:java}
 private boolean reportTransition(RegionStateNode regionNode, ServerStateNode 
serverNode,
    TransitionCode state, long seqId, long procId) throws IOException {
    ServerName serverName = serverNode.getServerName();
    TransitRegionStateProcedure proc = regionNode.getProcedure();
    if (proc == null) {
      return false;
    }
    proc.reportTransition(master.getMasterProcedureExecutor().getEnvironment(), 
regionNode,
      serverName, state, seqId, procId);
    return true;
  } {code}
If regionNode doesn't have any procedure, the master just logs it and doesn't 
throw any error to RPC. 

 

Think of a case when MasterFailover is happening and the new Active master only 
initialized the TRSP and CloseRegionProcedure. Now aborting Master has 
stale/false data. If the transition report comes to the aborting master, not 
rejecting this report is causing the procedure to stuck. 

 

Logs for more understanding 

{{{}2024{}}}{{{}-{}}}{{{}06{}}}{{{}-{}}}{{{}20{}}}{{ 
}}{{{}04{}}}{{{}:{}}}{{{}45{}}}{{{}:{}}}{{{}05{}}}{{{},{}}}{{{}576{}}}{{ 
}}{{ERROR}}{{{} 
[{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}write{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}3{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}0{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}61000{}}}{{{}]
 {}}}{{{}master{}}}{{{}.{}}}{{{}HMaster{}}}{{ }}{{-}}{{ ***** 
}}{*}{{{}ABORT{}}}{{{}ING{}}}{{ }}{{master}}{{{} 
server{}}}{{{}4{}}}{{{}-{}}}{{{}1{}}}{*}{{{},{}}}{{{}61000{}}}{{{},{}}}{{{}1715413775736{}}}{{{}:
 {}}}{{Failed}}{{ }}{{to}}{{ }}{{record}}{{ }}{{region}}{{ }}{{server}}{{ 
}}{{as}}{{ }}{{started}}{{ *****}}

{{{}2024{}}}{{{}-{}}}{{{}06{}}}{{{}-{}}}{{{}20{}}}{{ 
}}{{{}04{}}}{{{}:{}}}{{{}49{}}}{{{}:{}}}{{{}28{}}}{{{},{}}}{{{}893{}}}{{ 
}}{{DEBUG}}{{{} 
[{}}}{{{}aster{}}}{{{}/server{}}}{{{}5{}}}{{{}-{}}}{{{}1:{}}}{{{}61000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
 {}}}{{{}assignment{}}}{{{}.{}}}{{{}RegionStateStore{}}}{{ }}{{-}}{{ 
}}{{Load}}{{ }}{{{}hbase{}}}{{{}:{}}}{{{}meta{}}}{{ }}{{entry}}{{ 
}}{{{}region{}}}{{{}={}}}{{{}888a715d5926adbb89c985d8967f40d4{}}}{{{}, 
{}}}{{{}regionState{}}}{{{}={}}}{{{}OPEN{}}}{{{}, 
{}}}{{{}lastHost{}}}{{{}=server{}}}{{{}1{}}}{{{}-{}}}{{{}119{}}}{{{},{}}}{{{}61020{}}}{{{},{}}}{{{}1717560166420{}}}{{{},
 
{}}}{{{}regionLocation{}}}{{{}=server{}}}{{{}1{}}}{{{}-{}}}{{{}119{}}}{{{},{}}}{{{}61020{}}}{{{},{}}}{{{}1717560166420{}}}{{{},
 {}}}{{{}openSeqNum{}}}{{{}={}}}{{{}34892620{}}}

{{{}024-06-20 04:49:51,886 INFO [PEWorker-22] procedure2.ProcedureExecutor - 
Initialized subprocedures=[{pid={*}16276416{*}, ppid=16276108, 
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure 
table=RIMBS.UPLOADER_JOB_DETAILS, region=888a715d5926adbb89c985d8967f40d4, 
UNASSIGN}]  ({*}on server{*}{}}}{*}{{5-1}}{*}{{{}*)*{}}}

{{2024-06-20 04:49:52,022 INFO [PEWorker-40] procedure2.ProcedureExecutor - 
Initialized subprocedures=[\{pid=16276470, ppid=16276416, state=RUNNABLE; 
CloseRegionProcedure 888a715d5926adbb89c985d8967f40d4, 
server=server1-119,61020,1717560166420}] ({*}on server5-1){*}}}

 

*RS logs for closing* 

{{{}2024{}}}{{{}-{}}}{{{}06{}}}{{{}-{}}}{{{}20{}}}{{ 
}}{{{}04{}}}{{{}:{}}}{{{}49{}}}{{{}:{}}}{{{}52{}}}{{{},{}}}{{{}267{}}}{{ 
}}{{INFO}}{{{} 
[{}}}{{{}_{}}}{{{}REGION{}}}{{{}-{}}}{{{}regionserver{}}}{{{}/server{}}}{{{}1{}}}{{{}-{}}}{{{}119{}}}{{{}:{}}}{{{}61020{}}}{{{}-{}}}{{{}2{}}}{{{}]
 {}}}{{{}handler{}}}{{{}.{}}}{{{}UnassignRegionHandler{}}}{{ }}{{-}}{{ 
}}{{Close}}{{ }}{{888a715d5926adbb89c985d8967f40d4}}

2024-06-20 04:49:52,267 DEBUG [_REGION-regionserver/server1-119:61020-2] 
regionserver.HRegion - Closing 888a715d5926adbb89c985d8967f40d4, disabling 
compactions & flushes

{{{}2024{}}}{{{}-{}}}{{{}06{}}}{{{}-{}}}{{{}20{}}}{{ 
}}{{{}04{}}}{{{}:{}}}{{{}49{}}}{{{}:{}}}{{{}52{}}}{{{},{}}}{{{}354{}}}{{ 
}}{{INFO}}{{{} 
[{}}}{{{}_{}}}{{{}REGION{}}}{{{}-{}}}{{{}regionserver{}}}{{{}/server{}}}{{{}1{}}}{{{}-{}}}{{{}119{}}}{{{}:{}}}{{{}61020{}}}{{{}-{}}}{{{}2{}}}{{{}]
 {}}}{{{}regionserver{}}}{{{}.{}}}{{{}HRegion{}}}{{ }}{{-}}{{ }}{{Closed}}{{{} 
TABLE,{}}}{{{}KW{}}}{{{}\{}}}{{{}x00na240{}}}{{{}-{}}}{{{}app1{}}}{{{}-{}}}{{{}16{}}}{{{}\{}}}{{{}x00{}}}{{{}/{}}}{{{}Events{}}}{{{}-{}}}{{{}120620231740{}}}{{{}\{}}}{{{}x00{}}}{{{}MARKER{}}}{{{}-{}}}{{{}Events{}}}{{{},{}}}{{{}1702619592612{}}}{{{}.{}}}{{{}888a715d5926adbb89c985d8967f40d4.{}}}

 

*Logs on aborting active Hmaster*

{{{}2024{}}}{{{}-{}}}{{{}06{}}}{{{}-{}}}{{{}20{}}}{{ 
}}{{{}04{}}}{{{}:{}}}{{{}49{}}}{{{}:{}}}{{{}52{}}}{{{},{}}}{{{}355{}}}{{ 
}}{{WARN}}{{{} 
[{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}write{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}1{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}0{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}61000{}}}{{{}]
 {}}}{{{}assignment{}}}{{{}.{}}}{{{}AssignmentManager{}}}{{ }}{{-}}{{ 
}}*{{No}}{{ }}{{matching}}{{ }}{{procedure}}{{ }}{{found}}*{{ }}{{for}}{{{} 
server{}}}{{{}1{}}}{{{}-{}}}{{{}119{}}}{{{},{}}}{{{}61020{}}}{{{},{}}}{{{}1717560166420{}}}{{
 }}{{transition}}{{ }}{{on}}{{ }}{{{}state{}}}{{{}={}}}{{{}OPEN{}}}{{{}, 
{}}}{{{}location{}}}{{{}=server{}}}{{{}1{}}}{{{}-{}}}{{{}119{}}}{{{},{}}}{{{}61020{}}}{{{},{}}}{{{}1717560166420{}}}{{{},
 
{}}}{{{}table{}}}{{{}={}}}{{{}RIMBS{}}}{{{}.{}}}{{{}UPLOADER{}}}{{{}_{}}}{{{}JOB{}}}{{{}_{}}}{{{}DETAILS{}}}{{{},
 {}}}{{{}region{}}}{{{}={}}}{{{}888a715d5926adbb89c985d8967f40d4{}}}{{ 
}}{{to}}{{ }}{{CLOSED ( }}host = server{*}4-1{*} , 
{*}hbaseMasterLogFile{*}{{{}){}}}

{{}}

{{}}

{{}}


> Aborting Active HMaster is not rejecting reportRegionStateTransition if 
> procedure is initialised by next Active master
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-28690
>                 URL: https://issues.apache.org/jira/browse/HBASE-28690
>             Project: HBase
>          Issue Type: Bug
>          Components: proc-v2
>    Affects Versions: 2.5.8
>            Reporter: Umesh Kumar Kumawat
>            Priority: Major
>
> A {{CloseRegionProcedure on master request the RS to close the region and 
> after closing the region RS reports RegionStateTransition 
> back([here|https://github.com/apache/hbase/blob/d1015a68ed9f94d74668abd37edefd32f5e9305b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1853]).
>  On recieving the report, master checks if }}regionNode has any procedure 
> assigned to it 
> ([code|https://github.com/apache/hbase/blob/d1015a68ed9f94d74668abd37edefd32f5e9305b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1294]).
>  
>  
> {code:java}
>  private boolean reportTransition(RegionStateNode regionNode, ServerStateNode 
> serverNode,
>     TransitionCode state, long seqId, long procId) throws IOException {
>     ServerName serverName = serverNode.getServerName();
>     TransitRegionStateProcedure proc = regionNode.getProcedure();
>     if (proc == null) {
>       return false;
>     }
>     
> proc.reportTransition(master.getMasterProcedureExecutor().getEnvironment(), 
> regionNode,
>       serverName, state, seqId, procId);
>     return true;
>   } {code}
> If regionNode doesn't have any procedure, the master just logs it and doesn't 
> throw any error to RPC. 
>  
> Think of a case when MasterFailover is happening and the new Active master 
> only initialized the TRSP and CloseRegionProcedure. Now aborting Master has 
> stale/false data. If the transition report comes to the aborting master, not 
> rejecting this report is causing the procedure to stuck. 
>  
> *Logs for more understanding* 
> active master server4-1 failing
> {{}}
> {noformat}
> 2024-06-20 04:45:05,576 ERROR 
> [iority.RWQ.Fifo.write.handler=3,queue=0,port=61000] master.HMaster - ***** 
> ABORTING master server4-1,61000,1715413775736: Failed to record region server 
> as started *****{noformat}
> {{}}
>  
>  
> *logs of new active master server5-1*
>  
> {noformat}
> 2024-06-20 04:49:28,893 DEBUG [aster/server5-1:61000:becomeActiveMaster] 
> assignment.RegionStateStore - Load hbase:meta entry 
> region=888a715d5926adbb89c985d8967f40d4, regionState=OPEN, 
> lastHost=server1-119,61020,1717560166420, 
> regionLocation=server1-119,61020,1717560166420, openSeqNum=34892620
> 024-06-20 04:49:51,886 INFO [PEWorker-22] procedure2.ProcedureExecutor - 
> Initialized subprocedures=[{pid=16276416, ppid=16276108, 
> state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure 
> table=RIMBS.UPLOADER_JOB_DETAILS, region=888a715d5926adbb89c985d8967f40d4, 
> UNASSIGN}]  (on server5-1)
> 2024-06-20 04:49:52,022 INFO [PEWorker-40] procedure2.ProcedureExecutor - 
> Initialized subprocedures=[{pid=16276470, ppid=16276416, state=RUNNABLE; 
> CloseRegionProcedure 888a715d5926adbb89c985d8967f40d4, 
> server=server1-119,61020,1717560166420}] (on server5-1){noformat}
>  
>  
> *RS logs for closing* 
> {noformat}
> 2024-06-20 04:49:52,267 INFO [_REGION-regionserver/server1-119:61020-2] 
> handler.UnassignRegionHandler - Close 888a715d5926adbb89c985d8967f40d4
> 2024-06-20 04:49:52,267 DEBUG [_REGION-regionserver/server1-119:61020-2] 
> regionserver.HRegion - Closing 888a715d5926adbb89c985d8967f40d4, disabling 
> compactions & flushes
> 2024-06-20 04:49:52,354 INFO [_REGION-regionserver/server1-119:61020-2] 
> regionserver.HRegion - Closed 
> TABLE,KW\x00na240-app1-16\x00/Events-120620231740\x00MARKER-Events,1702619592612.888a715d5926adbb89c985d8967f40d4.
> {noformat}
> *Logs of report on aborting active Hmaster*
> {noformat}
> 2024-06-20 04:49:52,355 WARN 
> [iority.RWQ.Fifo.write.handler=1,queue=0,port=61000] 
> assignment.AssignmentManager - No matching procedure found for 
> server1-119,61020,1717560166420 transition on state=OPEN, 
> location=server1-119,61020,1717560166420, table=RIMBS.UPLOADER_JOB_DETAILS, 
> region=888a715d5926adbb89c985d8967f40d4 to CLOSED ( host = server4-1 , 
> hbaseMasterLogFile){noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to