Jianshi, I have seen many solutions to importing this kind of data:
1. Pre-splitting regions (I did not try this) 2. Using a map reduce job to create HFiles instead of putting individual rows into the database (instructions here: http://hbase.apache.org/book/arch.bulk.load.html 3. Modifying the row key to not be monotonic I went with the third solution by pre-prending a random integer before the other fields in my composite row key ( "<random int>_<key field 1>_<key field 2>Š.") When you make any changes, you can verify it is working by viewing the Hbase web interface (port 60010 on the hbase master) to see the requests per second on the various region servers. Thank you, Steven Magana-Zook On 9/5/14 9:14 AM, "Jianshi Huang" <[email protected]> wrote: >Thanks Ted, I'll try to do a major compact. > >Hi Steven, > >Yes, most of my rows are hashed to make it randomly distributed, but one >column family has monotonically increasing rowkeys, and it's used for >recording sequence of events. > >Do you have a solution how to bulk import this kind of data? > >Jianshi > > > >On Sat, Sep 6, 2014 at 12:00 AM, Magana-zook, Steven Alan < >[email protected]> wrote: > >> Hi Jianshi, >> >> What are the field(s) in your row key? If your row key is monotonically >> increasing then you will be sending all of your requests to one region >> server. Even after the region splits, all new entries will keep >>punishing >> one server (the region responsible for the split containing the new >>keys). >> >> See these articles that may help if this is indeed your issue: >> 1. http://hbase.apache.org/book/rowkey.design.html >> 2. >> >>http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-inc >>re >> asing-values-are-bad/ >> >> Regards, >> Steven Magana-Zook >> >> >> >> >> >> >> On 9/5/14 8:54 AM, "Jianshi Huang" <[email protected]> wrote: >> >> >Hi JM, >> > >> >What do you mean by the 'destination cluster'? The files are in the >>same >> >Hadoop/HDFS cluster where HBase is running. >> > >> >Do you mean do the bulk importing on HBase Master node? >> > >> > >> >Jianshi >> > >> > >> >On Fri, Sep 5, 2014 at 11:18 PM, Jean-Marc Spaggiari < >> >[email protected]> wrote: >> > >> >> Hi Jianshi, >> >> >> >> You might want to upload the file on the destination cluster first >>and >> >>then >> >> re-run your bulk load from there. That way the transfer time will >>not be >> >> taken into consideration for the timeout size the files will be >>local. >> >> >> >> JM >> >> >> >> >> >> 2014-09-05 11:15 GMT-04:00 Jianshi Huang <[email protected]>: >> >> >> >> > I'm importing 2TB of generated HFiles to HBase and I constantly get >> >>the >> >> > following errors: >> >> > >> >> > Caused by: >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop >>>>.h >> >>base.RegionTooBusyException): >> >> > org.apache.hadoop.hbase.RegionTooBusyException: failed to get a >>lock >> >>in >> >> > 60000 ms. >> >> > >> >> > >> >> >> >>>>regionName=grapple_edges_v2,ff000000,1409817320781.6d2955c780b39523de73 >>>>3f >> >>3565642d96., >> >> > server=xxxxx.xxx.xxx,60020,1404854700728 >> >> > at >> >> > >>org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5851) >> >> > at >> >> > >>org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5837) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.regionserver.HRegion.startBulkRegionOperation(H >>>>Re >> >>gion.java:5795) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.jav >>>>a: >> >>3543) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.jav >>>>a: >> >>3525) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegio >>>>nS >> >>erver.java:3277) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2 >>>>.c >> >>allBlockingMethod(ClientProtos.java:28863) >> >> > at >> >> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008) >> >> > at >> >>org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcSc >>>>he >> >>duler.java:160) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcSche >>>>du >> >>ler.java:38) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler >>>>.j >> >>ava:110) >> >> > at java.lang.Thread.run(Thread.java:724) >> >> > >> >> > at >> >> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1498) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java >>>>:1 >> >>684) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation. >>>>ca >> >>llBlockingMethod(RpcClient.java:1737) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$B >>>>lo >> >>ckingStub.bulkLoadHFile(ClientProtos.java:29276) >> >> > at >> >> > >> >> > >> >> >> >>>>org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUti >>>>l. >> >>java:1548) >> >> > ... 11 more >> >> > >> >> > >> >> > What makes the region too busy? Is there a way to improve it? >> >> > >> >> > Does that also mean some part of my data are not correctly >>imported? >> >> > >> >> > >> >> > Thanks, >> >> > >> >> > -- >> >> > Jianshi Huang >> >> > >> >> > LinkedIn: jianshi >> >> > Twitter: @jshuang >> >> > Github & Blog: http://huangjs.github.com/ >> >> > >> >> >> > >> > >> > >> >-- >> >Jianshi Huang >> > >> >LinkedIn: jianshi >> >Twitter: @jshuang >> >Github & Blog: http://huangjs.github.com/ >> >> > > >-- >Jianshi Huang > >LinkedIn: jianshi >Twitter: @jshuang >Github & Blog: http://huangjs.github.com/
