HI J-D,
I have tried to force unassign it with shell command 'unassign
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.', true',
but it didn't help.
The error message still appears after executing that command, and it looks
like just same as before:
2011-08-04 09:56:41,005 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e.
(offlining)
2011-08-04 09:56:41,005 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e. but
it is not currently assigned anywhere
2011-08-04 09:57:09,847 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1312422839848
2011-08-04 09:57:09,848 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
2011-08-04 09:57:09,849 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
(offlining)
2011-08-04 09:57:09,850 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
which is already pending close but forcing an additional close
2011-08-04 09:57:09,851 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
usedHeap=11121, maxHeap=15983) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
2011-08-04 09:57:49,089 INFO
org.apache.hadoop.hbase.master.handler.DisableTableHandler: Offlining 1
regions.
2011-08-04 09:58:53,791 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 1 region(s) in transition:
{aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1312423029850}
2011-08-04 09:58:55,724 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 5422 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-08-04 10:00:19,848 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1312423029850
2011-08-04 10:00:19,848 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
2011-08-04 10:00:19,850 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
(offlining)
2011-08-04 10:00:19,850 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
which is already pending close but forcing an additional close
2011-08-04 10:00:19,852 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
usedHeap=11121, maxHeap=15983) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
Let me know if I have made any mistake.
Thanks,
Yi
On Tue, Aug 2, 2011 at 4:38 AM, Jean-Daniel Cryans <[email protected]>wrote:
> You need to force unassign it using the shell.
>
> J-D
>
> On Mon, Aug 1, 2011 at 12:33 AM, Yi Liang <[email protected]> wrote:
> > We're running hbase 0.90.3. For some unknown reason, we now can't disable
> > one table because its first region can't be unassigned.
> >
> > The log message looks like following and it repeats endlessly:
> >
> > 2011-07-25 13:27:23,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> > timed out:
> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > state=PENDING_CLOSE, ts=1311571458790
> > 2011-07-25 13:27:23,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> > PENDING_CLOSE for too long, running forced unassign again on
> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > 2011-07-25 13:27:23,746 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment
> of
> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > (offlining)
> > 2011-07-25 13:27:23,746 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > which is already pending close but forcing an additional close
> > 2011-07-25 13:27:23,748 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
> > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
> > usedHeap=11121, maxHeap=15983) returned
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.NotServingRegionException: Received close for
> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we
> are
> > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
> > 2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
> > running balancer because 1 region(s) in transition:
> >
> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > state=PENDING_CLOSE, ts=1311571643746}
> > 2011-07-25 13:28:55,308 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor:
> > Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
> > 2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
> > client connection, connectString=192.168.201.7:2181,192.168.201.6:2181,
> > 192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
> > 2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> > connection to server /192.168.201.7:2181
> > 2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
> > connection established to test07.xxx.com/192.168.201.7:2181, initiating
> > session
> > 2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
> > establishment complete on server test07.xxx.com/192.168.201.7:2181,
> > sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
> > 2011-07-25 13:30:17,583 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> > Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
> > max=2147483647 rows
> > 2011-07-25 13:30:17,627 INFO
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Closed zookeeper sessionid=0x3314bcb3ae38799
> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x3314bcb3ae38799 closed
> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn: EventThread
> > shut down
> > 2011-07-25 13:30:33,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> > timed out:
> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > state=PENDING_CLOSE, ts=1311571643746
> > 2011-07-25 13:30:33,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> > PENDING_CLOSE for too long, running forced unassign again on
> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> >
> > The problem also blocks the region balancer.
> >
> > Can I fix it without restarting the whole hbase cluster?
> >
> > Thanks,
> > Yi
> >
>
