---------- Forwarded message ----------
From: stack <[email protected]>
To: [email protected]
Date: Mon, 17 Aug 2009 14:21:47 -0700
Subject: Re: NoServerForRegionException, TableNotFoundException and
WrongRegionException
Please update to the head of 0.19 trunk, or better update to 0.20 trunk --
espeically if you are testing. Issues described below have been addressed.
How many regions do you have in your table? Are all going to one
regionserver because you only have one region?
Yours,
St.Ack
On Mon, Aug 17, 2009 at 12:19 PM, Marc Limotte <[email protected]>
wrote:
I'm seeing a nice variety of Exceptions from HBase and could use some
pointers about what to do next.
This is a new map/reduce program, updating about 550k rows with around a
dozen columns on a very small cluster (only 4 nodes... as we're still
testing and it doesn't have to support production yet). Hbase Version
0.19.1.
I ran the job and it seems to make some progress, and then dies after
several hours, reporting "NoServerForRegionException: No server address
listed in .META. for region TABLEX,,1250526695078". I retried it a few
times with the same result. I also noticed that the load is not well
balanced, all requests seemed to be going to one node. I adjust
hadoop-site.xml with the addition of these two entries:
<name>hbase.hregion.max.filesize</name>
<value>33554432</value>
<name>hbase.client.retries.number</name>
<value>5</value>
And restarted hbase (and hadoop to be safe). Re-ran and got the same
error
in the M/R job.
*I thought I'd try dropping the table, since it's a new table and I can
recreate it. But that gives another exception:
*
hbase(main):002:0> disable 'TABLEX'
NativeException: org.apache.hadoop.hbase.TableNotFoundException:
org.apache.hadoop.hbase.TableNotFoundException: TABLEX
at
org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:129)
at
org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:70)
at
org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
at
org.apache.hadoop.hbase.master.TableOperation.process(TableOperation.java:143)
at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:691)
...
*And now I see this exception in the HBase logs:
*
org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
out
of range for HRegion .META.,,1250280235390, startKey='',
getEndKey()='TABLEX,,1250219949252',
row='TABLEX,840.56098.0544,1250526661861'
at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1788)
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1844)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1912)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1244)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1216)
...
*As a test, tried a "count"...
*
hbase(main):007:0* count 'TABLEX'
NativeException:
org.apache.hadoop.hbase.client.NoServerForRegionException:
No server address listed in .META. for region TABLEX,,1250526695078
from org/apache/hadoop/hbase/client/HConnectionManager.java:548:in
`locateRegionInMeta'
from org/apache/hadoop/hbase/client/HConnectionManager.java:478:in
`locateRegion'
from org/apache/hadoop/hbase/client/HConnectionManager.java:440:in
`locateRegion'
from org/apache/hadoop/hbase/client/HTable.java:114:in `<init>'
from org/apache/hadoop/hbase/client/HTable.java:97:in `<init>'
from sun/reflect/NativeConstructorAccessorImpl.java:-2:in
`newInstance0'
...
*Also saw a thread somewhere that suggested doing a major compaction.
Did
that. It returns almost immediately. Not sure if that's normal or
not...
no perceivable impact from doing this, though.*
hbase(main):013:0> major_compact '.META.'
0 row(s) in 0.0220 seconds
hbase(main):014:0>
Not sure what else to try? Is there a way to force removal of the table
in
question? Is there something else I should be looking at?
Marc
