Meta is giving out the wrong address for a region? Do a scan of .META. It might be easier dumping the scan into a file so you can grep around:
echo "scan '.META.'" | ./bin/hbase shell --format-width=300 &> /tmp/meta.txt Grep in here for the region that contains the row you are looking for. What does it have for info:server? Go to that regionserver (UI or log). Is it carrying the region? If not, thats what the WRE is about. For same region, grep its name in master log (hopefully you have DEBUG enabled). Whats its history? Could it have been assigned to one server and then another? If so, close the region in both places. Type 'tools' in the shell to see doc. on "close_region" command. You can pass it server to pass the close message to. Close in both places. If its a double-assignment issue, our name for above phenomeon, suggest you upgrade to 0.20.1. It has at least one pointed fix for this scenario (HBASE-1878). St.Ack On Wed, Nov 4, 2009 at 12:35 PM, Joost Ouwerkerk <[email protected]>wrote: > HBase has started throwing WrongRegionExceptions at me when trying to > access > certain regions. I'm guessing that the META table has somehow gone out of > sync with reality. I've tried compacting and I've tried restarting, but > the > problem does not go away. The errors are always on the same regions. Has > anyone else seen this and succeeded at getting their table back into > working > order? > > *Example get:* > > org.apache.hadoop.hbase.regionserver.WrongRegionException: > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row > out > of range for HRegion > > crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084, > > startKey='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F', > > getEndKey()='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fhermosa-beach\x2Fall-cuisines\x2Ftags\x2Foutdoor-dining\x2F', > > row='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F' > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1522) > at > > org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1554) > at > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1622) > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2278) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > *Example put: > * > put 'crawled_pages','r: > http://com.xxxx.yyyy/restaurants/all-areas/inglewood/all-cuisines/', > 'curi:test','test' > NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: > Trying to contact region server Some server, retryOnlyOne=true, index=0, > islastrow=true, tries=4, numtries=5, i=0, listsize=1, > > region=crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084 > for region > > crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084, > row > > 'r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F', > but failed after 5 attempts. > Exceptions: > > from org/apache/hadoop/hbase/client/HConnectionManager.java:1119:in > `process' > from org/apache/hadoop/hbase/client/HConnectionManager.java:1200:in > `processBatchOfRows' > from org/apache/hadoop/hbase/client/HTable.java:605:in `flushCommits' > from org/apache/hadoop/hbase/client/HTable.java:470:in `put' > from org/apache/hadoop/hbase/client/HTable.java:1761:in `commit' > from org/apache/hadoop/hbase/client/HTable.java:1742:in `commit' > from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0' > from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke' > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' > from java/lang/reflect/Method.java:597:in `invoke' > from org/jruby/javasupport/JavaMethod.java:298:in > `invokeWithExceptionHandling' > from org/jruby/javasupport/JavaMethod.java:259:in `invoke' > from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call' > from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call' > from org/jruby/ast/CallOneArgNode.java:57:in `interpret' > from org/jruby/ast/NewlineNode.java:104:in `interpret' >
