That region was offline a while probably because it was taking a while for split daughters to come online. If you do a Get in shell for the start row, is it available now? Grep in master for 1291950247501. See if you can figure out a history of regions carrying this row. See if you see something like a region split that perhaps was taking a while. Go take a look at the regionserver log where split was happening (The master in its logs should report the location of the plit). What was going on on this region? Was it struggling? GC pause? Swapping?
NotServingRegionException can be part of 'normal' operation. The server will throw the client this exception as signal that it should recalibrate -- i.e. go back to .META. to find a regions new location -- because the region has moved because of split or crash, etc. if the NSREs go on for too long, they turn from DEBUG into ERRORs and client fails. If loading on cluster is intensive, it can take a while for regions to re-online. There could be another issue in the way of the region re-onlining. Grepping around in the logs as per above should give a clue. St.Ack On Thu, Dec 9, 2010 at 10:00 PM, Tao Xie <[email protected]> wrote: > hi, all > > I met this exception when I doing intensive insertions using YCSB. Anybody > give me some clues on this? I use hbase 0.20.6. > > com.yahoo.ycsb.DBException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server -- nothing found, no 'location' returned, > tableName=usertable, reload=true -- for region , row 'user1001412274', but > failed after 11 attempts. > Exceptions: > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > > at com.yahoo.ycsb.db.HBaseClient.cleanup(Unknown Source) > at com.yahoo.ycsb.DBWrapper.cleanup(Unknown Source) > at com.yahoo.ycsb.ClientThread.run(Unknown Source) > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying > to contact region server -- nothing found, no 'location' returned, > tableName=usertable, reload=true -- for region , row 'user1001412274', but > failed after 11 attempts. > Exceptions: > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in .META. for region usertable,,1291950247501 > > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocationForRowWithRetries(HConnectionManager.java:1095) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.access$200(HConnectionManager.java:240) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.getRegionName(HConnectionManager.java:1191) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1168) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1238) > at > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666) > ... 3 more >
