Thanks Stack. Yes, upping the xceivercount fixed the previous issue.
The system is more stable than ever. I used the "spelling" you
provided. Maybe add to FAQ?
grep the master I see the region but unfortunately, I did not have
DEBUG on at the time. I added DEBUG to the log4j settings. After
restarting I saw this:
hbase-root-master-dev-hadoop1.brandaffinity.net.log:2008-11-20
10:28:31,265 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current
assignment of metrics,Game Face|News,1226620888277 is not valid:
serverInfo: address: 10.10.10.51:60020, startcode: 1227205702706,
load: (requests: 4 regions: 1), passed startCode: 1227124620094,
storedInfo.startCode: 1227205702706, unassignedRegions: false,
pendingRegions: false
hbase-root-master-dev-hadoop1.brandaffinity.net.log:2008-11-20
10:28:48,156 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region metrics,Game Face|News,1226620888277 to server
10.10.10.51:60020
I ran the jobcounter again and it was successful. Looks like HBase
recovered on it's own.
Thanks again,
Dru
On Nov 20, 2008, at 10:11 AM, stack wrote:
Says its 'closed'.
Look in master log for references to this region. See if it was
ever online. If it was, see which host it was up on; e.g. do
something like "grep 'metrics,Game Face|News,1226620888277'"
master.log". Look at its RegionInfo -- should be output in master
log if you have DEBUG enabled. Look at the encoded name. See that
the region exists still in hdfs by doing "./bin/hadoop fs -lsr
HBASE_ROOTDIR/metrics/ENCODED_NAME. Next add '-v metaScanner' to
weed the metaScanner output when grepping master log to see trail of
events as they pertain to this particular region. Might give you a
clue as to what happened. When you scan the .META., does this
region appear at all? Is there a 'hole' in the .META. where this
region should be? We may have to reinsert if so. To do this, will
need old value for HRI.
Did the upping of xceivercount work for your previous issue Dru?
St.Ack
Dru Jensen wrote:
I have a table that has 20+ million rows. I tried to run
rowcounter MR process against it but one of the task attempts fail
on the following exception:
java.io.IOException: java.io.IOException: Region metrics,Game Face|
News,1226620888277 closed
at
org
.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:
1336)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:
1175)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl
.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC
$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
I checked .meta. and I don't see this region online. Should it be?
Any ideas how to fix it?
Thanks,
Dru
