Hello,
I'm running hbase 0.20.5, and seeing Puts() fail repeatedly when trying to
insert a specific item into the database.
Client side I see:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true, tries=9,
numtries=10, i=0, listsize=1,
region=filestore,bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b,1279604506836
for region filestore,
I then looked up which node was hosting the given region
(filestore,bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b) on
the gui, found the following debug message in the regionserver log:
2010-08-06 14:23:47,414 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Batch puts interrupted at
index=0 because:Requested row out of range for HRegion
filestore,bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b,1279604506836,
startKey='bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b',
getEndKey()='be0bc7b3f8bc2a30910b9c758b47cdb730a4691e93f92abb857a2dcc7aefa633',
row='be1681910b02db5da061659c2cb08f501a135c2f065559a37a1761bf6e203d1d'
Which appears to be coming from:
/regionserver/HRegionServer.java:1786: LOG.debug("Batch puts interrupted
at index=" + i + " because:" +
Which is coming from:
./java/org/apache/hadoop/hbase/regionserver/HRegion.java:1658: throw new
WrongRegionException("Requested row out of range for " +
This happens repeatedly on a specific item over at least a day or so, even when
not much is happening with the cluster.
As far as I can tell, it looks like the logic to select the correct region for
a given row is wrong. The row is indeed not in the correct range (at least from
what I can tell of the exception thrown), and the check in HRegion.java:1658:
/** Make sure this is a valid row for the HRegion */
private void checkRow(final byte [] row) throws IOException {
if(!rowIsInRange(regionInfo, row)) {
Is correctly rejecting the Put().
So it appears the error would be somewhere in:
HRegion.java:1550:
private void put(final Map<byte [],List<KeyValue>> familyMap,
boolean writeToWAL) throws IOException {
Which appears to be the actual guts of the insert operation.
However, I don't know enough about the design of HRegions to really decipher
this method. I'll dig into it more, but I thought it might be more efficient
just to ask you guys first.
Any ideas?
I can update to 0.20.6, but I don't see any fixed jira's on 0.20.6 that seem
related.. I could be wrong. I'm not sure what I should do next. Any more
information you guys need?
Note that I am inserting file into the database, and using it's sha256sum as
the key. And the file that is failing does indeed have a sha that corresponds
to the key in the message above (and is out of range).
Take care,
-stu
