Hi Todd, Thanks. The problem was I used the patch for multi column family bulk upload first and then added the LZO modifications. The code in the trunk for creating writers is different from this so my code changes will not be equivalent for the trunk and the mutli column family patch.
Any chances the multi column family patch will make it to the trunk soon? Anyways will generate a patch against the trunk as well and attach it to the ticket. Thanks and regards, - Ashish On Wed, 26 Jan 2011 19:50:06 -0800 Todd Lipcon <[email protected]> wrote: > On Wed, Jan 26, 2011 at 2:43 AM, Ashish Shinde <[email protected]> > wrote: > > > Hi, > > > > I am using 0.90.0 candidate_3 from > > http://people.apache.org/~stack/hbase-0.90.0-candidate-3/ > > > > and have patched it to run multi column family bulk upload from > > ticket > > > > https://issues.apache.org/jira/browse/HBASE-1861 > > > > Also created the ticket > > https://issues.apache.org/jira/browse/HBASE-3474 > > to handle this. > > > > I modified the code but not sure how to generate a patch. The multi > > column family code looks to be on 0.92.0 branch which I > > can't find in hbase svn repo. how do I create the patch. > > > > 0.92 is just "trunk" in the svn repo - so you can generate your patch > against that. > > Or, if we want we can consider this a bug fix / compatible > improvement, and we can apply it to both 0.90 (for 0.90.1) and trunk. > But, we'll need a patch for trunk as well. > > -Todd > > On Tue, 25 > > Jan 2011 10:00:00 +0530 Ashish Shinde <[email protected]> wrote: > > > > > Hi, > > > > > > Yup after some digging I got to HFileOutputFormat and was > > > relieved to know that it does support compression. Was able to > > > add code to set compression based on the column family's > > > compression setting. > > > > > > Will create a ticket and submit the patch after some more testing > > > and going over the coding guidelines. My code looks a little > > > hacky because I am passing the family specific compression > > > algorithm name as "," delimited single configuration item. I > > > figure that Configuration should have a method to return all key > > > values where key's match a pattern. Maybe there are better ways > > > to do this. Will get this into the ticket. > > > > > > Thanks and regards, > > > - Ashish > > > > > > On Mon, 24 Jan 2011 11:12:06 -0800 > > > Todd Lipcon <[email protected]> wrote: > > > > > > > On Mon, Jan 24, 2011 at 9:50 AM, Stack <[email protected]> wrote: > > > > > > > > > In HFileOutputFormat it says this near top: > > > > > > > > > > // Invented config. Add to hbase-*.xml if other than > > > > > default compression. > > > > > final String compression = conf.get("hfile.compression", > > > > > Compression.Algorithm.NONE.getName()); > > > > > > > > > > You might try messing with this config? > > > > > > > > > > > > > And would be great to file (and provide a patch for) a JIRA that > > > > automatically sets this based on the HTableDescriptor when > > > > you're loading into an existing table! > > > > > > > > -Todd > > > > > > > > > > > > > On Sun, Jan 23, 2011 at 9:38 PM, Ashish Shinde > > > > > <[email protected]> wrote: > > > > > > Hi, > > > > > > > > > > > > I have been importing data to hbase 0.90.0 using the code > > > > > > from the bulk uploader (ImportTsv.java). The table has LZO > > > > > > compression set, however unless major compaction is run the > > > > > > table it does not get compressed. > > > > > > > > > > > > Is there a way to compress the table as the bulk uploader > > > > > > creates the HFile. This is important for us because we don't > > > > > > want to have a burst increase in our disk usage. > > > > > > > > > > > > Thanks and regards, > > > > > > - Ashish > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
