A batch update holds the puts for a single row, are you really updating 5k cols per row?
generally i turn autoflushing off, use a 1-12mb client buffer. even so, you can see 'cant find address' or 'not serving region exception' during a transition of regions. check the region historian for the region in question to see if there were moves. one other thing, are you getting exceptions throw from table.commit() ? the existing client will retry to smooth over region transitions, so you can get exceptions in the logfile but not actually be thrown from table.commit(). One thing I do is do my uploads using java programs and map reduces - this ensures maximal performance and lets me do automated retries. 24m rows should import in no time. I can do about 500m rows in 4 hours... good luck! -ryan On Fri, May 15, 2009 at 4:53 PM, llpind <[email protected]> wrote: > > Hey All, > > Finally finished. But didn't complete (~ between 24,770,000 - 25,760,000 > records processed). Got the following exception before java upload app > died: > > org.apache.hadoop.hbase.client.NoServerForRegionException: No server > address > listed in .META. for region > > I believe it has to do with "Problem: DFS instability and/or regionserver > lease timeouts" (http://wiki.apache.org/hadoop/Hbase/Troubleshooting). So > i > was going to try to put some JVM flags in for GC. Also, when creating the > table, is there any column options I should use that will help in upload? > > When i try to get results from within HBase shell it gives the following > exception: > > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: > <TABLE_NAME>,<ROWID>,1242429308026 > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2071) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1534) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) > > > from org/apache/hadoop/hbase/client/HConnectionManager.java:858:in > `getRegionServerWithRetries' > from org/apache/hadoop/hbase/client/HTable.java:608:in `getRow' > from org/apache/hadoop/hbase/client/HTable.java:583:in `getRow' > from org/apache/hadoop/hbase/client/HTable.java:478:in `getRow' > from org/apache/hadoop/hbase/client/HTable.java:425:in `getRow' > from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0' > from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke' > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' > from java/lang/reflect/Method.java:597:in `invoke' > from org/jruby/javasupport/JavaMethod.java:298:in > `invokeWithExceptionHandling' > from org/jruby/javasupport/JavaMethod.java:259:in `invoke' > from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call' > from org/jruby/runtime/callsite/CachingCallSite.java:280:in > `cacheAndCall' > from org/jruby/runtime/callsite/CachingCallSite.java:116:in `call' > from org/jruby/ast/CallOneArgNode.java:57:in `interpret' > from org/jruby/ast/LocalAsgnNode.java:120:in `interpret' > ... 120 levels... > from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in > `call' > from org/jruby/internal/runtime/methods/CompiledMethod.java:216:in > `call' > from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in > `call' > from org/jruby/runtime/callsite/CachingCallSite.java:260:in > `cacheAndCall' > from org/jruby/runtime/callsite/CachingCallSite.java:75:in `call' > from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:441:in > `__file__' > from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:-1:in `__file__' > from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:-1:in `load' > from org/jruby/Ruby.java:564:in `runScript' > from org/jruby/Ruby.java:467:in `runNormally' > from org/jruby/Ruby.java:340:in `runFromMain' > from org/jruby/Main.java:214:in `run' > from org/jruby/Main.java:100:in `run' > from org/jruby/Main.java:84:in `main' > from /home/hadoop/hbase19/bin/../bin/hirb.rb:338:in `get' > > > > I am open to any other ideas. Should my batch update be smaller than > 10,000? ...5,000? I'd like to make any adjustments before i start this > long > upload again. Thanks. > -- > View this message in context: > http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23568944.html > Sent from the HBase User mailing list archive at Nabble.com. > >
