Looks like the issue of HBASE-4064. Hope it will be fixed ASAP. But for now, how can I fix the issue?
Due to the region in transition forever, the hbase balancer doesn't work any more. I have tried to kill and restart master alone, but it doesn't help. Do we have to restart the whole cluster for this region? Thanks, Yi On Thu, Aug 4, 2011 at 10:07 AM, Yi Liang <[email protected]> wrote: > HI J-D, > > I have tried to force unassign it with shell command 'unassign > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.', true', > but it didn't help. > > The error message still appears after executing that command, and it looks > like just same as before: > 2011-08-04 09:56:41,005 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of > region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e. > (offlining) > 2011-08-04 09:56:41,005 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign > region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e. but > it is not currently assigned anywhere > 2011-08-04 09:57:09,847 INFO > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition > timed out: > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > state=PENDING_CLOSE, ts=1312422839848 > 2011-08-04 09:57:09,848 INFO > org.apache.hadoop.hbase.master.AssignmentManager: Region has been > PENDING_CLOSE for too long, running forced unassign again on > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > 2011-08-04 09:57:09,849 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > (offlining) > 2011-08-04 09:57:09,850 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > which is already pending close but forcing an additional close > 2011-08-04 09:57:09,851 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName= > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237, > usedHeap=11121, maxHeap=15983) returned > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Received close for > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa > 2011-08-04 09:57:49,089 INFO > org.apache.hadoop.hbase.master.handler.DisableTableHandler: Offlining 1 > regions. > 2011-08-04 09:58:53,791 DEBUG org.apache.hadoop.hbase.master.HMaster: Not > running balancer because 1 region(s) in transition: > {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > state=PENDING_CLOSE, ts=1312423029850} > 2011-08-04 09:58:55,724 DEBUG > org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5422 catalog row(s) > and gc'd 0 unreferenced parent region(s) > 2011-08-04 10:00:19,848 INFO > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition > timed out: > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > state=PENDING_CLOSE, ts=1312423029850 > 2011-08-04 10:00:19,848 INFO > org.apache.hadoop.hbase.master.AssignmentManager: Region has been > PENDING_CLOSE for too long, running forced unassign again on > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > 2011-08-04 10:00:19,850 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > (offlining) > 2011-08-04 10:00:19,850 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. > which is already pending close but forcing an additional close > 2011-08-04 10:00:19,852 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName= > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237, > usedHeap=11121, maxHeap=15983) returned > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Received close for > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa > > Let me know if I have made any mistake. > > Thanks, > Yi > > On Tue, Aug 2, 2011 at 4:38 AM, Jean-Daniel Cryans <[email protected]>wrote: > >> You need to force unassign it using the shell. >> >> J-D >> >> On Mon, Aug 1, 2011 at 12:33 AM, Yi Liang <[email protected]> wrote: >> > We're running hbase 0.90.3. For some unknown reason, we now can't >> disable >> > one table because its first region can't be unassigned. >> > >> > The log message looks like following and it repeats endlessly: >> > >> > 2011-07-25 13:27:23,745 INFO >> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition >> > timed out: >> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > state=PENDING_CLOSE, ts=1311571458790 >> > 2011-07-25 13:27:23,745 INFO >> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been >> > PENDING_CLOSE for too long, running forced unassign again on >> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > 2011-07-25 13:27:23,746 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment >> of >> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > (offlining) >> > 2011-07-25 13:27:23,746 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign >> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > which is already pending close but forcing an additional close >> > 2011-07-25 13:27:23,748 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName= >> > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237, >> > usedHeap=11121, maxHeap=15983) returned >> > org.apache.hadoop.hbase.NotServingRegionException: >> > org.apache.hadoop.hbase.NotServingRegionException: Received close for >> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we >> are >> > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa >> > 2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster: >> Not >> > running balancer because 1 region(s) in transition: >> > >> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > state=PENDING_CLOSE, ts=1311571643746} >> > 2011-07-25 13:28:55,308 DEBUG >> org.apache.hadoop.hbase.master.CatalogJanitor: >> > Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s) >> > 2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating >> > client connection, connectString=192.168.201.7:2181,192.168.201.6:2181, >> > 192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection >> > 2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening >> socket >> > connection to server /192.168.201.7:2181 >> > 2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket >> > connection established to test07.xxx.com/192.168.201.7:2181, initiating >> > session >> > 2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session >> > establishment complete on server test07.xxx.com/192.168.201.7:2181, >> > sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000 >> > 2011-07-25 13:30:17,583 DEBUG >> org.apache.hadoop.hbase.client.MetaScanner: >> > Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for >> > max=2147483647 rows >> > 2011-07-25 13:30:17,627 INFO >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> > Closed zookeeper sessionid=0x3314bcb3ae38799 >> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session: >> > 0x3314bcb3ae38799 closed >> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn: >> EventThread >> > shut down >> > 2011-07-25 13:30:33,745 INFO >> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition >> > timed out: >> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > state=PENDING_CLOSE, ts=1311571643746 >> > 2011-07-25 13:30:33,745 INFO >> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been >> > PENDING_CLOSE for too long, running forced unassign again on >> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. >> > >> > The problem also blocks the region balancer. >> > >> > Can I fix it without restarting the whole hbase cluster? >> > >> > Thanks, >> > Yi >> > >> > >
