[
https://issues.apache.org/jira/browse/PIG-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798922#action_12798922
]
Gerrit Jansen van Vuuren commented on PIG-1117:
-----------------------------------------------
OK I can make the changes and resubmit the patch if there are no objections?
You're right that the reason more having this separate was that the UDFs that
do not need the Hive Jars are not affected by this.
The hive jars are not downloaded using ivy but just plain old ant, I've tried
to find a way to do using Ivy/Maven but the Hive jars are not published to any
Maven repository that I know off. So the ant build.xml downloads the tar.gz
file directly from apache, unzip it and copy the hive_exec.jar to the lib
folder from where its added to the class path. This is only done if the
hive_exec.jar is not present.
> Pig reading hive columnar rc tables
> -----------------------------------
>
> Key: PIG-1117
> URL: https://issues.apache.org/jira/browse/PIG-1117
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.7.0
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Gerrit Jansen van Vuuren
> Fix For: 0.7.0
>
> Attachments: HiveColumnarLoader.patch, HiveColumnarLoaderTest.patch,
> PIG-1117.patch, PIG-117-v.0.6.0.patch, PIG-117-v.0.7.0.patch
>
>
> I've coded a LoadFunc implementation that can read from Hive Columnar RC
> tables, this is needed for a project that I'm working on because all our data
> is stored using the Hive thrift serialized Columnar RC format. I have looked
> at the piggy bank but did not find any implementation that could do this.
> We've been running it on our cluster for the last week and have worked out
> most bugs.
>
> There are still some improvements to be done but I would need like setting
> the amount of mappers based on date partitioning. Its been optimized so as to
> read only specific columns and can churn through a data set almost 8 times
> faster with this improvement because not all column data is read.
> I would like to contribute the class to the piggybank can you guide me in
> what I need to do?
> I've used hive specific classes to implement this, is it possible to add this
> to the piggy bank build ivy for automatic download of the dependencies?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
