Thanks for trying. Looks like that region is now gone (split is my guess). Check the master log for mentions of this region to see its history. Can you correlate the client failure with an event on this region in master log? It looks like client was being pig-headed fixated on the parent of a split. You could check your table is healthy? Run the rowcounter program to make sure no holes in table?
St.Ack On Fri, Aug 14, 2009 at 12:41 PM, llpind <[email protected]> wrote: > > hbase(main):003:0> get '.META.', 'TestTable,0001749889,1250092414985', > {COLUMNS =>'info'} > 09/08/14 12:28:10 DEBUG client.HConnectionManager$TableServers: Cache hit > for row <> in tableName .META.: location server 192.168.0.196:60020, > location region name .META.,,1 > NativeException: java.lang.NullPointerException: null > from org/apache/hadoop/hbase/client/HTable.java:789:in `get' > from org/apache/hadoop/hbase/client/HTable.java:769:in `get' > from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0' > from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke' > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' > from java/lang/reflect/Method.java:597:in `invoke' > from org/jruby/javasupport/JavaMethod.java:298:in > `invokeWithExceptionHandling' > from org/jruby/javasupport/JavaMethod.java:259:in `invoke' > from org/jruby/java/invokers/InstanceMethodInvoker.java:30:in `call' > from org/jruby/runtime/callsite/CachingCallSite.java:30:in `call' > from org/jruby/ast/CallManyArgsNode.java:59:in `interpret' > from org/jruby/ast/LocalAsgnNode.java:123:in `interpret' > from org/jruby/ast/NewlineNode.java:104:in `interpret' > from org/jruby/ast/IfNode.java:112:in `interpret' > from org/jruby/ast/NewlineNode.java:104:in `interpret' > from org/jruby/ast/IfNode.java:114:in `interpret' > ... 115 levels... > from > home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb#start:-1:in > `call' > from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in > `call' > from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in > `call' > from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in > `call' > from org/jruby/runtime/callsite/CachingCallSite.java:253:in > `cacheAndCall' > from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call' > from > home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb.rb:487:in > `__file__' > from > home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb.rb:-1:in > `load' > from org/jruby/Ruby.java:577:in `runScript' > from org/jruby/Ruby.java:480:in `runNormally' > from org/jruby/Ruby.java:354:in `runFromMain' > from org/jruby/Main.java:229:in `run' > from org/jruby/Main.java:110:in `run' > from org/jruby/Main.java:94:in `main' > from /home/hadoop/hbase-0.20.0/bin/../bin/hirb.rb:384:in `get' > from (hbase):4hbase(main):004:0> get '.META.', > 'TestTable,0001749889,1250092414985' > 09/08/14 12:28:13 DEBUG client.HConnectionManager$TableServers: Cache hit > for row <> in tableName .META.: location server 192.168.0.196:60020, > location region name .META.,,1 > COLUMN CELL > historian:assignment timestamp=1250108456441, value=Region assigned > to server server195,60020,12501083767 > 79 > historian:compaction timestamp=1250109313965, value=Region > compaction completed in 35sec > historian:open timestamp=1250108459484, value=Region opened > on > server : server195 > historian:split timestamp=1250092447915, value=Region split > from: TestTable,0001634945,1250035163 > 027 > info:regioninfo timestamp=1250109315260, value=REGION => {NAME > => 'TestTable,0001749889,125009241 > 4985', STARTKEY => '0001749889', ENDKEY => > '0001866010', ENCODED => 1707908074, O > FFLINE => true, TABLE => {{NAME => 'TestTable', > FAMILIES => [{NAME => 'info', VER > SIONS => '3', COMPRESSION => 'NONE', TTL => > '2147483647', BLOCKSIZE => '65536', I > N_MEMORY => 'false', BLOCKCACHE => 'true'}]}} > > > ===================================== > > stack-3 wrote: > > > > Is that region offline? > > > > Do a: > > > > hbase> get ".META.", "TestTable,0001749889,1250092414985", {COLUMNS => > > "info"}. > > > > If so, can you get its history so we can figure how it went offline? (See > > region history in UI or grep it in master logs?) > > > > St.Ack > > > > > > On Fri, Aug 14, 2009 at 9:55 AM, llpind <[email protected]> wrote: > > > >> > >> Hey Stack, I tried the following command: > >> > >> hadoop-0.20.0/bin/hadoop jar hbase-0.20.0/hbase-0.20.0-test.jar > >> randomWrite > >> 10 > >> > >> running a map/reduce job, it failed with the following exceptions in > each > >> node: > >> > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > >> contact > >> region server Some server for region , row '0001753186', but failed > after > >> 11 > >> attempts. > >> Exceptions: > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline: > >> TestTable,0001749889,1250092414985 > >> > >> at > >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocationForRowWithRetries(HConnectionManager.java:995) > >> at > >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1025) > >> at > >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584) > >> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450) > >> at > >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest.testRow(PerformanceEvaluation.java:497) > >> at > >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:406) > >> at > >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:627) > >> at > >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:194) > >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >> at > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356) > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > >> at org.apache.hadoop.mapred.Child.main(Child.java:170) > >> > >> ================================================================ > >> > >> This appears to be this issue: > >> > >> http://issues.apache.org/jira/browse/HBASE-1603 > >> > >> > >> Has this been fixed in .20? Thanks. > >> > >> > >> > >> stack-3 wrote: > >> > > >> > On Wed, Aug 12, 2009 at 8:58 AM, llpind <[email protected]> > wrote: > >> > > >> >> > >> >> Playing with the HBase perfomanceEval Class, but it seems to take a > >> long > >> >> time to run “sequentialWrite 2” (~20 minutes). If I simply emulate 1 > >> >> clients in a simple program, I can do 1 Million Puts in about 3 > >> minutes > >> >> (non > >> >> mapred). The sequential write is writing 2 million with 2 clients. > >> >> Please > >> >> help in understanding how to use the performanceEvaluation Class. > >> >> > >> > > >> > If the number of clients is > 1, unless you add the '--nomapred' (sp?) > >> > argument, PE launches a mapreduce program of N tasks. Each task puts > >> up > >> a > >> > client writing 1M rows (IIRC). Try N where N == number_of_map_slots > >> and > >> > see > >> > what that does? N == 2 probably won't tell you much. You could also > >> set > >> > an > >> > N > 1 and use the '--nomapred'. This will run PE clients in a > distinct > >> > thread. For small numbers of N, this can put up heavier loading than > >> MR > >> > with its setup and teardown cost. > >> > > >> > St.Ack > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24975031.html > >> Sent from the HBase User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24977400.html > Sent from the HBase User mailing list archive at Nabble.com. > >
