HI Ankit ， But I try to restart the RS several times , it’s does NOT to auto fix the RIT region .
> On 5 Sep 2019, at 10:00, Ankit Singhal <[email protected]> wrote: > > The only problem with this approach is that if RS responsible for closing > the region did not > flush the data properly then explicitly setting it to CLOSED in hbase:meta > can result in data loss until > you killed the RS so that the data can be replayed through WAL. > > Thanks, > Ankit Singhal > > On Wed, Sep 4, 2019 at 6:15 PM Syni Guo <[email protected]> wrote: > >> >> HI , >> >> Thanks for your suggection , I used a extreme method solved the issue . >> >> 1. Change the RIT region 'info:state’ value to 'CLOSED' of the meta table >> ‘hbase:mata’ >> 2. Switch over the master node . >> >> >>> On 4 Sep 2019, at 00:05, Wellington Chevreuil < >> [email protected]> wrote: >>> >>> The "unassign" procs for each of those regions are waiting on a lock >> that's >>> currently held by another proc whose id is 21075720: >>>> >>>> 2019-09-02 19:51:29,365 INFO [PEWorker-1] >>>> procedure.MasterProcedureScheduler: Waiting on xlock for >>>> pid=21097026...held by pid=21075720 >>>> >>> >>> You need to figure out why this pid=21075720 is stuck while it keeps the >>> lock(s) needed by the two "unassign" procs. You can figure out which proc >>> is this via Web UI or hbase shell list_procedures command. From past >>> experience, I would bet this is a rogue "disable_table" command who had >>> timed out long ago. A meta scan for this 'alpha_daas:device_data_details' >>> table regions would also help determine if this indeed the case: >>> $ echo "scan 'hbase:meta'" | hbase shell | grep >>> "alpha_daas:device_data_details" >>> >>> Em ter, 3 de set de 2019 às 00:12, Syni Guo <[email protected]> >>> escreveu: >>> >>>> Hi Ankit, >>>> >>>> How can i find out the block session or process and kill them. it does >> not >>>> work even reboot the RS >>>> >>>> >>>> 获取 Outlook for Android<https://aka.ms/ghei36> >>>> >>>> ________________________________ >>>> From: Ankit Singhal <[email protected]> >>>> Sent: Monday, September 2, 2019 11:47:06 PM >>>> To: [email protected] <[email protected]> >>>> Subject: Re: Region in RIT (CLOSING) , How to fix it ? >>>> >>>> you may check the RS logs of tx-220-70-27.h.chinabank.com.cn for >>>> exceptions(like sync failure >>>> while writing marker in the WAL during closure etc ) which are keeping >>>> these regions stuck in the >>>> CLOSING state and you may also try killing this server for >>>> ServerCrashProcedure to take care of this. >>>> >>>> Regards, >>>> Ankit Singhal >>>> >>>> On Mon, Sep 2, 2019 at 6:52 AM Jean-Marc Spaggiari < >>>> [email protected]> >>>> wrote: >>>> >>>>> Hi Syni, >>>>> >>>>> Have you tried using HBCK2? >>>>> >>>>> JMS >>>>> >>>>> Le lun. 2 sept. 2019 07 h 57, Syni Guo <[email protected]> a >>>> écrit >>>>> : >>>>> >>>>>> >>>>>> >>>>>> Hbase version : 2.1.3 >>>>>> >>>>>> >>>>>> There are 2 region in RIT (CLOSING) , How to fix it ? , I try to >>>> unassign >>>>>> it ,but timeout failed . >>>>>> >>>>>> >>>>>> hbase(main):032:0> unassign '444405785869685e6ec948c03c2076b8',true >>>>>> >>>>>> ERROR: Call id=13259, waitTime=10009, rpcTimeout=10000 >>>>>> >>>>>> For usage try 'help "unassign”' >>>>>> >>>>>> >>>>>> >>>>>> Logs; >>>>>> >>>>>> 2019-09-02 19:50:54,453 WARN [ProcExecTimeout] >>>>>> assignment.AssignmentManager: STUCK Region-In-Transition rit=CLOSING, >>>>>> location=tx-220-70-27.h.chinabank.com.cn,60020,1567410218074, >>>>>> table=alpha_daas:device_data_details, >>>>>> region=444405785869685e6ec948c03c2076b8 >>>>>> 2019-09-02 19:50:54,453 WARN [ProcExecTimeout] >>>>>> assignment.AssignmentManager: STUCK Region-In-Transition rit=CLOSING, >>>>>> location=tx-220-70-27.h.chinabank.com.cn,60020,1567410218074, >>>>>> table=alpha_daas:poi_unicom_stat, >>>> region=42de1052551760e45cb7ba2684d586f8 >>>>>> >>>>>> >>>>>> >>>>>> 2019-09-02 19:51:29,365 INFO [PEWorker-1] >>>>>> procedure.MasterProcedureScheduler: Waiting on xlock for pid=21097026, >>>>>> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure >>>>>> table=alpha_daas:device_data_details, >>>>>> region=444405785869685e6ec948c03c2076b8, server= >>>>>> tx-220-70-27.h.chinabank.com.cn,60020,1567410218074 held by >>>> pid=21075720 >>>>>> 2019-09-02 19:51:39,581 INFO [PEWorker-11] >>>>>> procedure.MasterProcedureScheduler: Waiting on xlock for pid=21097027, >>>>>> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure >>>>>> table=alpha_daas:device_data_details, >>>>>> region=444405785869685e6ec948c03c2076b8, server= >>>>>> tx-220-70-27.h.chinabank.com.cn,60020,1567410218074 held by >>>> pid=21075720 >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >> >>
