[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487179#comment-13487179
]
Joseph Adler commented on PIG-3015:
-----------------------------------
Just reading through the discussion on the user list.
I'll check out trunk, refactor/rename as needed, make sure it passes existing
tests, fix bugs, then submit the patches. That will probably take me a few days
to do.
Additionally, I'd like to get a few things correct the first time.
Specifically, I'm trying to figure out how to deal with the plethora of
possible options for load/store functions. I want to make sure that I cover all
the important use cases regarding schemas. Here's the list that I came up with:
LoadFunc:
(1) Read the schema from the input file(s)
(a) Just pick the schema from the most recent file
(b) Check all the files to make sure the schemas are compatible
(2) Use a schema manually provided by the user
StoreFunc:
(1) Automatically translate the Pig schema to an Avro Schema
(2) Use a schema manually provided by the user
(a) Allow the user to name the records and name space
(b) Automatically pick a record and namespace name
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
