I'm seeing a nice variety of Exceptions from HBase and could use some
pointers about what to do next.
This is a new map/reduce program, updating about 550k rows with around a
dozen columns on a very small cluster (only 4 nodes... as we're still
testing and it doesn't have to support production yet). Hbase Version
0.19.1.
I ran the job and it seems to make some progress, and then dies after
several hours, reporting "NoServerForRegionException: No server address
listed in .META. for region TABLEX,,1250526695078". I retried it a few
times with the same result. I also noticed that the load is not well
balanced, all requests seemed to be going to one node. I adjust
hadoop-site.xml with the addition of these two entries:
<name>hbase.hregion.max.filesize</name>
<value>33554432</value>
<name>hbase.client.retries.number</name>
<value>5</value>
And restarted hbase (and hadoop to be safe). Re-ran and got the same error
in the M/R job.
*I thought I'd try dropping the table, since it's a new table and I can
recreate it. But that gives another exception:
*
hbase(main):002:0> disable 'TABLEX'
NativeException: org.apache.hadoop.hbase.TableNotFoundException:
org.apache.hadoop.hbase.TableNotFoundException: TABLEX
at
org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:129)
at
org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:70)
at
org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
at
org.apache.hadoop.hbase.master.TableOperation.process(TableOperation.java:143)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:691)
...
*And now I see this exception in the HBase logs:
*
org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
of range for HRegion .META.,,1250280235390, startKey='',
getEndKey()='TABLEX,,1250219949252',
row='TABLEX,840.56098.0544,1250526661861'
at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1788)
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1844)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1912)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1244)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1216)
...
*As a test, tried a "count"...
*
hbase(main):007:0* count 'TABLEX'
NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException:
No server address listed in .META. for region TABLEX,,1250526695078
from org/apache/hadoop/hbase/client/HConnectionManager.java:548:in
`locateRegionInMeta'
from org/apache/hadoop/hbase/client/HConnectionManager.java:478:in
`locateRegion'
from org/apache/hadoop/hbase/client/HConnectionManager.java:440:in
`locateRegion'
from org/apache/hadoop/hbase/client/HTable.java:114:in `<init>'
from org/apache/hadoop/hbase/client/HTable.java:97:in `<init>'
from sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
...
*Also saw a thread somewhere that suggested doing a major compaction. Did
that. It returns almost immediately. Not sure if that's normal or not...
no perceivable impact from doing this, though.*
hbase(main):013:0> major_compact '.META.'
0 row(s) in 0.0220 seconds
hbase(main):014:0>
Not sure what else to try? Is there a way to force removal of the table in
question? Is there something else I should be looking at?
Marc
