On Nov 30, 2009, at 12:18 PM, Dmitriy Ryaboy wrote:
That's awesome, I've been itching to do that but never got around to
it..
Garrit, do you have any benchmarks on read speeds?
I don't know about putting this in piggybank, as it carries with it
pretty
significant dependencies, increasing the size of the jar and making it
difficult for users to don't need it to build piggybank in the first
place.
We might want to consider some other contrib for it -- maybe a "misc"
contrib that would have indivudual ant targets for these kinds of
compatibility submissions?
Does it have to increase the size of the piggybank jar? Instead of
including hive in our piggybank jar, which I agree would be bad, can
we just say that if you want to use this function you need to provide
the appropriate hive jar yourself? This way we could use ivy to pull
the jars and build piggybank.
I'm not really wild about creating a new section of contrib just for
functions that have heavier weight requirements.
Alan.
-D
On Mon, Nov 30, 2009 at 3:09 PM, Olga Natkovich <ol...@yahoo-
inc.com> wrote:
Hi Garrit,
It would be great if you could contribute the code. The process is
pretty simple:
- Open a JIRA that describes what the loader does and that you would
like to contribute it to the Piggybank.
- Submit the patch that contains the loader. Make sure it has unit
tests
and javadoc.
On this is done, one of the committers will review and commit the
patch.
More details on how to contribute are in
http://wiki.apache.org/pig/PiggyBank.
Olga
-----Original Message-----
From: Gerrit van Vuuren [mailto:gvanvuu...@specificmedia.com]
Sent: Friday, November 27, 2009 2:42 AM
To: pig-dev@hadoop.apache.org
Subject: Pig reading hive columnar rc tables
Hi,
I've coded a LoadFunc implementation that can read from Hive
Columnar RC
tables, this is needed for a project that I'm working on because
all our
data is stored using the Hive thrift serialized Columnar RC format. I
have looked at the piggy bank but did not find any implementation
that
could do this. We've been running it on our cluster for the last week
and have worked out most bugs.
There are still some improvements to be done but I would need like
setting the amount of mappers based on date partitioning. Its been
optimized so as to read only specific columns and can churn through a
data set almost 8 times faster with this improvement because not all
column data is read.
I would like to contribute the class to the piggybank can you guide
me
in what I need to do?
I've used hive specific classes to implement this, is it possible
to add
this to the piggy bank build ivy for automatic download of the
dependencies?
Thanks,
Gerrit Jansen van Vuuren
