http://pastebin.com/mU8tpuiD there it is.
On Tue, Sep 28, 2010 at 12:03 PM, Stack <[email protected]> wrote: > Can you get history of this region from master log and pastebin it > (paste result of a grep of 84162fcdf083fe4736c39571223cb029 to > pastebin)? > St.Ack > > On Tue, Sep 28, 2010 at 10:53 AM, Jack Levin <[email protected]> wrote: >> I am using rest and getting following errors on some insertions: >> >> >> 2010-09-28 10:51:07,215 ERROR >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested >> row out of range for HRegion >> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029., >> startKey='samp754.png', getEndKey()='screenshot205.png', >> row='sf6c.jpg' >> at >> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367) >> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464) >> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027) >> >> And from REST log: >> >> >> 2010-09-28 10:50:26,308 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Failed >> execution of all on mtab5.prod.imageshack.com:60020 >> org.apache.hadoop.hbase.regionserver.WrongRegionException: >> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested >> row out of range for HRegion >> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029., >> startKey='samp754.png', getEndKey()='screenshot205.png', >> row='sf6c.jpg' >> at >> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367) >> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464) >> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027) >> >> at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown >> Source) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >> at >> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.translateException(HConnectionManager.java:1552) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithoutRetries(HConnectionManager.java:1162) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1530) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1528) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> It looks like even after restarting REST I get the same error, could >> this been that zookeepers cached the wrong location of the region and >> its not on the same regionserver? Are there other methods to debug? >> >> -Jack >> >
