Hey guys,
There seems to be something wrong on trunk... I used to have long map-reduce
jobs, but now they are failing, unable to commit:
2009-04-04 01:17:09,279 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
locateRegionInMeta attempt 5 of 10 failed; retrying after sleep of 8000
java.io.IOException: HRegionInfo was null or empty in .META.
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:566)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:484)
... etc
Basically mappers get stuck up on commits and make no progress, mapred kills
them, done.
I've spent some time banging at it - made sure that ulimit -n is good, set
the ipc handler limit to 30, cranked down the number of maps I'm doing,
etc. To no avail.
At least I figured out how to debug hadoop jobs a bit.
Anyone have thoughts?
