I am using rest and getting following errors on some insertions:
2010-09-28 10:51:07,215 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
row out of range for HRegion
img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
startKey='samp754.png', getEndKey()='screenshot205.png',
row='sf6c.jpg'
at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
And from REST log:
2010-09-28 10:50:26,308 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Failed
execution of all on mtab5.prod.imageshack.com:60020
org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
row out of range for HRegion
img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
startKey='samp754.png', getEndKey()='screenshot205.png',
row='sf6c.jpg'
at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.translateException(HConnectionManager.java:1552)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithoutRetries(HConnectionManager.java:1162)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1530)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1528)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
It looks like even after restarting REST I get the same error, could
this been that zookeepers cached the wrong location of the region and
its not on the same regionserver? Are there other methods to debug?
-Jack
