Apache9 commented on PR #8584: URL: https://github.com/apache/hbase/pull/8584#issuecomment-5487364435
> > I think this is an optimization for not splitting unnecessary wal edits, but why it will lead to merge stuck? After opening a region, we should have removed all the recovered.edits files? > > Thanks @Apache9 for looking into it. Below is incident brief explanation. I've also raised [PR8583](https://github.com/apache/hbase/pull/8583) to reopen parent region on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS linked to this incident. > > What Happened: > > 1. Region 112d9f08 was gracefully moved from rs-132 → rs-81 at 16:36:34 UTC. > 2. The close on rs-132 had already flushed the region's edits, and rs-81 opened the region with openSeqNum=4997750282, establishing that the edits up to that point were durable. > 3. ~25 seconds later, rs-132 was declared dead as part of the broader graceful RS drain. > 4. The WAL split worker (rs-34) subsequently created a recovered.edits file for this region containing edit seqId=4997750280. > 5. This edit was already durable in the HFiles. The recovered edit file was effectively stale/orphaned, but its existence was not recognized as harmless. > > Where it got stuck: > > At 18:32:33 UTC, MergeTableRegionsProcedure (pid=46990253) unassigned the region successfully. During MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, the procedure saw the recovered.edits file and failed the check. The merge procedure retained the region lock and did not recover automatically, leaving the region in CLOSED/RIT for ~48m 56s. The RIT only cleared after the HMaster failover caused the procedure to replay and fresh top-level ASSIGN procedures reopened the regions. Then basically there are two problems. On 5, we should remove the recovered.edits when opening the region, of course a failure of removing should not be considered as a critical issue. And in MergeTableRegionsProcedure, when we have a recovered.edits file, we should check if the edits are all below the persistent seqNum, if so we are OK to remove the directory and go on. We can do these changes in a separated issue. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
