[
https://issues.apache.org/jira/browse/HBASE-26518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-26518:
-------------------------------
Summary: Concurrent assign and SCP may cause regionserver abort (was:
Concurrent assign the same region may cause regionserver abort)
> Concurrent assign and SCP may cause regionserver abort
> ------------------------------------------------------
>
> Key: HBASE-26518
> URL: https://issues.apache.org/jira/browse/HBASE-26518
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 1.2.0
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: master-error-logs.png
>
>
> When concurrent assign a region, the RS may abort when reporting region state
> to master encounters transit region state error,
>
> {code:java}
> 2021-11-26 04:40:54,689 INFO
> [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65]
> regionserver.HRegionServer: Post open deploy tasks for
> growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.
> 2021-11-26 04:40:54,689 INFO
> [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65]
> regionserver.HRegionServer: Failed to transition {ENCODED =>
> ee1df69da9c7fb7698d2972f81d40b65, NAME =>
> 'growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.',
> STARTKEY => 'cc', ENDKEY => 'cd'} to OPENED:
> ee1df69da9c7fb7698d2972f81d40b65 is not pending open on
> zf-data-hbase250.mt,16020,1637872783437
> 2021-11-26 04:40:54,689 FATAL
> [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65]
> regionserver.HRegionServer: ABORTING region server
> zf-data-hbase250.mt,16020,1637872783437: Exception running
> postOpenDeployTasks; region=ee1df69da9c7fb7698d2972f81d40b65
> java.io.IOException: Failed to report opened region to master:
> growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2387)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:327)
> {code}
> In the master log, we can see that the region is being assigned by both the
> bulk assigner and the SCP.
> !master-error-logs.png|width=600,height=140!
> This steps can reproduce the problem:
> # thread 1 calls Assignment#assign to assign region A, and calls
> RegionStates#updateRegionsState and stops before the synchronize;
> # thread 2 calls RegionStates#serverOffline and gets the lock, it sees that
> region A is on the processed dead server and add region A to the
> regionsToOffline list(region A is not in the RIT cache now);
> # thread 2 prepare to execute RegionStates#regionOffline, and released the
> lock;
> # thread 1 get the lock, and add region A to the RIT cache, and also update
> the region state by RegionStates#putRegionState, set the region state from
> OFFLINE to PENDING_OPEN, then release the lock;
> # thread2 execute RegionStates#regionOffline, acquires the lock and sets
> the region state from PENDING_OPEN to OFFLINE;
> # the RS that region A assigned by thread 1 report region A should be
> transited to OPENED, but finds that region A last state is OFFLINE, error
> occures, the RS abort itself.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
