Hello Renaud: You've seen this documentation for bulk import in 0.20.x: http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk? (Make sure you are on 0.20.6).
In TRUNK bulk import was revamped. Its all fancy and robust now. See http://hbase.apache.org/docs/r0.89.20100726/bulk-loads.html In both versions a partitioner is required. In TRUNK the hadoop total order partitioner is brought local and should work for most key types. In 0.20.x you'd need to write your own. In 0.20.x, there is no support for incremental loading. It will only load a fresh table. Incremental is a feature of the TRUNK version. In 0.20.x, you use the loadtable.rb script. In TRUNK, you run a little java program. St.Ack On Thu, Sep 23, 2010 at 4:16 AM, Renaud Delbru <[email protected]> wrote: > Hi, > > we are trying to create a hbase table from scratch using map-reduce and > HFileOutputFormat. However, we haven't really find examples or tutorials on > how to do this, and there is some aspects which are still unclear for us. We > are using hbase 0.20.x. > > First, what is the correct way to use HFileOutputFormat and to create HFile > ? > We are simply using a map function which output <ImmutableBytesWritable > (key), Put (value)>, an identity reducer, and we configure the job to use > HFileOutputFormat as an output format class. > However, we have seen in hbase 0.89.x a more complex way to do it, involving > sorting (KeyValueSortReducer, or PutSortReducer) and a partitioner > (TotalOrderPartitioner). The HFileOutputFormat provides a convenience > method, configureIncrementalLoad, to automatically configure the hadoop job. > Is this method needed in our case ? Ir is this only necessary in the case > where the table already exists (incremental bulk load) ? > Do we have to reimplement this for 0.20.x ? > > Then, one time the table creation job is successful, how do we import the > hfiles into hbase ? Is it by using the hbase cli import command ? > > Thanks in advance for your answers, > Regards > -- > Renaud Delbru >
