Region is left unassigned after a split/rebalancing, throws NSRE ----------------------------------------------------------------
Key: HBASE-851 URL: https://issues.apache.org/jira/browse/HBASE-851 Project: Hadoop HBase Issue Type: Bug Affects Versions: 0.2.0 Reporter: Jean-Daniel Cryans Fix For: 0.19.0 Master log: {code} 2008-08-28 12:12:27,174 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 from 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:27,174 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: web_pages,http://www.salonskincare.co.uk/product_info.php/products_id/168,1219939934794 from 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:27,174 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 from 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server 192.168.1.95:60020 is overloaded. Server load: 8 avg: 7.0 <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing to reassign 1 regions. mostLoadedRegions has 8 regions in it. <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:27,175 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: web_pages,http://www.salonskincare.co.uk/product_info.php/products_id/168,1219939934794 open on 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:27,175 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1 <jdcryans> 2008-08-28 12:12:27,175 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row web_pages,http://www.salonskincare.co.uk/product_info.php/products_id/168,1219939934794 in region .META.,,1 with startcode 1219931259154 and server 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:30,352 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 from 192.168.1.95:60020 <jdcryans> 2008-08-28 12:1 <jdcryans> 2008-08-28 12:12:32,557 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 103, Num Servers: 15, Avg Load: 7.0 <jdcryans> 2008-08-28 12:12:34,093 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:34,093 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 open on 192.168.1.95:60020 <jdcryans> 2008-08-28 12:12:34,093 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1 <jdcryans> 2008-08-28 12:12:34,093 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 in region .META.,,1 with startcode 1219931259154 and server 192.168.1.95:60020 {code} HRS 192.168.1.95 {code} jdcryans> 2008-08-28 12:12:24,953 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,307 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794: [EMAIL PROTECTED] <jdcryans> 2008-08-28 12:12:27,307 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794: [EMAIL PROTECTED] <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1860667227/attribute <jdcryans> 2008-08-28 12:12:27,308 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> 2008-08-28 12:12:34,246 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([EMAIL PROTECTED], row => http://www.simplewebengines.com/, {column => attribute:traveliness, value => '...', column => attribute:processed_at, value => '...', column => attribute:content, value => '...', column => attribute:refs, value => '...', column => attribute:crawled_at, value => '...', column => att <jdcryans> ribute:html, value => '...', column => attribute:crawled, value => '...'}) from 192.168.1.96:50102: error: org.apache.hadoop.hbase.NotServingRegionException: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 <jdcryans> org.apache.hadoop.hbase.NotServingRegionException: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 NSRE for a hundred times {code} Restarting the cluster cleared the issue but this is a nasty bug. Proposed bandaid would be that if we have a NSRE after the retries, asked the master to scan the HRS to see if it's located somewhere else. If not, assign it somewhere. Finally update META. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.