Yeah I have looked at HBase. I do not think it meets my needs in this particular case. I want to efficiently do adhoc analysis on arbitrary subsets of columns using map reduce. Below are some of the reasons I feel HBase does not fit my needs.
To get the data in separate files, I would need to put each column in its own column family using a row id as the key. Each column family will end up as a seperate file. HBase will sort each column family independently, so if I had 100 columns I would be doing 100 times more sorting than I need to do. I believe all of this sorting would make insert rates really low. HBase supports an arbitrary number of columns per a row in a column family. To do this each row value has <col name>=<col value> pairs. For my case this is unessecary overhead as I would only have one column name per column family. It seems that when a map reduce job is run against HBase that it reads input through the HBase server. I suspect reading gzip files off local disk is much faster, but I am not sure. -----Original Message----- From: Ted Dunning [mailto:[EMAIL PROTECTED] Sent: Mon 3/10/2008 2:57 PM To: [email protected] Subject: Re: File Per Column in Hadoop Have you looked at hbase. It looks like you are trying to reimplement a bunch of it. On 3/10/08 11:01 AM, "Richard K. Turner" <[EMAIL PROTECTED]> wrote: > ... [storing data in columns is nice] ... I would also do the same for dir csv_file2. Does anyone know how to do this > in Hadoop?
