Is there a way to rebuild the META? I'm really hoping there's no data loss here, and it's just a question of META being out of sync with data... jo
On Wed, Nov 4, 2009 at 7:07 PM, Joost Ouwerkerk <[email protected]>wrote: > I investigated following your guidance, Stack. Unfortunately I am not > seeing evidence of double assignment. It looks more like a case of missing > assignment. There appear to be key ranges that are not represented in the > .META. table. So, I have a region that handles keys AAA to BBB, and the > next region handles DDD to EEE. Now when I try to access key CCC, I get > routed to the region that handles AAA to BBB, presumably because my key is > after AAA and before DDD. Then HRegion.checkRow fails because the requested > key is outside the region's range. > > Consider this error: > > org.apache.hadoop.hbase.regionserver.WrongRegionException: > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out > of range for HRegion > crawled_pages,r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fbasil-in-the-grove, > startKey > ='r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fbasil-in-the-grove', > getEndKey() > ='r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Feast-broward', > row > ='r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fhavana-hideout' > > As the error points out, the requested row is outside the range for the > region. In the .META. table, the next region starts at > 'r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fpashas-3'. The request row > falls after one region's End key, and before the next region's Start key. > > jo > > > On Wed, Nov 4, 2009 at 4:56 PM, stack <[email protected]> wrote: > >> Meta is giving out the wrong address for a region? Do a scan of .META. >> It >> might be easier dumping the scan into a file so you can grep around: >> >> echo "scan '.META.'" | ./bin/hbase shell --format-width=300 &> >> /tmp/meta.txt >> >> Grep in here for the region that contains the row you are looking for. >> What >> does it have for info:server? Go to that regionserver (UI or log). Is it >> carrying the region? If not, thats what the WRE is about. >> >> For same region, grep its name in master log (hopefully you have DEBUG >> enabled). >> >> Whats its history? Could it have been assigned to one server and then >> another? >> >> If so, close the region in both places. Type 'tools' in the shell to see >> doc. on "close_region" command. You can pass it server to pass the close >> message to. Close in both places. >> >> If its a double-assignment issue, our name for above phenomeon, suggest >> you >> upgrade to 0.20.1. It has at least one pointed fix for this scenario >> (HBASE-1878). >> >> St.Ack >> >> >> On Wed, Nov 4, 2009 at 12:35 PM, Joost Ouwerkerk <[email protected] >> >wrote: >> >> > HBase has started throwing WrongRegionExceptions at me when trying to >> > access >> > certain regions. I'm guessing that the META table has somehow gone out >> of >> > sync with reality. I've tried compacting and I've tried restarting, but >> > the >> > problem does not go away. The errors are always on the same regions. >> Has >> > anyone else seen this and succeeded at getting their table back into >> > working >> > order? >> > >> > *Example get:* >> > >> > org.apache.hadoop.hbase.regionserver.WrongRegionException: >> > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row >> > out >> > of range for HRegion >> > >> > >> crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084, >> > >> > >> startKey='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F', >> > >> > >> getEndKey()='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fhermosa-beach\x2Fall-cuisines\x2Ftags\x2Foutdoor-dining\x2F', >> > >> > >> row='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F' >> > at >> > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1522) >> > at >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1554) >> > at >> > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1622) >> > at >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2278) >> > at >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785) >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) >> > at >> > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> > at java.lang.reflect.Method.invoke(Method.java:597) >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) >> > at >> > >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) >> > >> > *Example put: >> > * >> > put 'crawled_pages','r: >> > http://com.xxxx.yyyy/restaurants/all-areas/inglewood/all-cuisines/', >> > 'curi:test','test' >> > NativeException: >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> > Trying to contact region server Some server, retryOnlyOne=true, index=0, >> > islastrow=true, tries=4, numtries=5, i=0, listsize=1, >> > >> > >> region=crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084 >> > for region >> > >> > >> crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084, >> > row >> > >> > >> 'r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F', >> > but failed after 5 attempts. >> > Exceptions: >> > >> > from org/apache/hadoop/hbase/client/HConnectionManager.java:1119:in >> > `process' >> > from org/apache/hadoop/hbase/client/HConnectionManager.java:1200:in >> > `processBatchOfRows' >> > from org/apache/hadoop/hbase/client/HTable.java:605:in `flushCommits' >> > from org/apache/hadoop/hbase/client/HTable.java:470:in `put' >> > from org/apache/hadoop/hbase/client/HTable.java:1761:in `commit' >> > from org/apache/hadoop/hbase/client/HTable.java:1742:in `commit' >> > from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0' >> > from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke' >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' >> > from java/lang/reflect/Method.java:597:in `invoke' >> > from org/jruby/javasupport/JavaMethod.java:298:in >> > `invokeWithExceptionHandling' >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke' >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call' >> > from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call' >> > from org/jruby/ast/CallOneArgNode.java:57:in `interpret' >> > from org/jruby/ast/NewlineNode.java:104:in `interpret' >> > >> > >
