Sorry I misspelled your name, Gerrit.

-D

On Mon, Nov 30, 2009 at 3:18 PM, Dmitriy Ryaboy <dvrya...@gmail.com> wrote:

> That's awesome, I've been itching to do that but never got around to it..
> Garrit, do you have any benchmarks on read speeds?
>
> I don't know about putting this in piggybank, as it carries with it pretty
> significant dependencies, increasing the size of the jar and making it
> difficult for users to don't need it to build piggybank in the first place.
> We might want to consider some other contrib for it -- maybe a "misc"
> contrib that would have indivudual ant targets for these kinds of
> compatibility submissions?
>
> -D
>
>
> On Mon, Nov 30, 2009 at 3:09 PM, Olga Natkovich <ol...@yahoo-inc.com>wrote:
>
>> Hi Garrit,
>>
>> It would be great if you could contribute the code. The process is
>> pretty simple:
>>
>> - Open a JIRA that describes what the loader does and that you would
>> like to contribute it to the Piggybank.
>> - Submit the patch that contains the loader. Make sure it has unit tests
>> and javadoc.
>>
>> On this is done, one of the committers will review and commit the patch.
>>
>> More details on how to contribute are in
>> http://wiki.apache.org/pig/PiggyBank.
>>
>> Olga
>>
>> -----Original Message-----
>> From: Gerrit van Vuuren [mailto:gvanvuu...@specificmedia.com]
>> Sent: Friday, November 27, 2009 2:42 AM
>> To: pig-dev@hadoop.apache.org
>> Subject: Pig reading hive columnar rc tables
>>
>> Hi,
>>
>>
>>
>> I've coded a LoadFunc implementation that can read from Hive Columnar RC
>> tables, this is needed for a project that I'm working on because all our
>> data is stored using the Hive thrift serialized Columnar RC format. I
>> have looked at the piggy bank but did not find any implementation that
>> could do this. We've been running it on our cluster for the last week
>> and have worked out most bugs.
>>
>>
>>
>> There are still some improvements to be done but I would need  like
>> setting the amount of mappers based on date partitioning. Its been
>> optimized so as to read only specific columns and can churn through a
>> data set almost 8 times faster with this improvement because not all
>> column data is read.
>>
>>
>>
>> I would like to contribute the class to the piggybank can you guide me
>> in what I need to do?
>>
>> I've used hive specific classes to implement this, is it possible to add
>> this to the piggy bank build ivy for automatic download of the
>> dependencies?
>>
>>
>>
>> Thanks,
>>
>>  Gerrit Jansen van Vuuren
>>
>>
>

Reply via email to