Do you know your keyspace roughly? Try creating a pre-split table with as many regions as you want reducers. St.Ack
On Wed, Sep 14, 2011 at 8:25 PM, rajesh balamohan <[email protected]> wrote: > ImportTSV internally uses HFileOutputFormat.configureIncrementalLoad(job, > table); > > However, for newly created tables there would not be any keys available. > Hence, it launches 1 reducer by default. > > Is there a way to increase the number of reducers for high volume imports > like 500+ GB. > > ~Rajesh.B > > On Thu, Sep 15, 2011 at 8:51 AM, rajesh balamohan > <[email protected]>wrote: > >> Hi All, >> >> ImportTSV is a great tool for bulk loading the data into HBASE. >> >> I have close to 500+GB of raw data which I would like to import into a >> newly created HTABLE. If I go ahead with ImportTSV, it creates only one >> reducer which is a bottleneck in terms of sorting and shuffling. >> >> Are there any other way, I can increase the number of reducers while doing >> bulk loads for new table?. >> >> ~Rajesh.B >> >
