-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8104/
-----------------------------------------------------------
(Updated Dec. 20, 2012, 5:24 p.m.)
Review request for pig and Cheolsoo Park.
Changes
-------
Several bug fixes. (A couple fixes to get the tests to run correctly, and a fix
to TrevniStorage so that the schema is correctly propagated to mappers/reducers
in the storefunc)
Description
-------
The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't support
Snappy compression.)
I rewrote AvroStorage from scratch to fix these issues. In early tests, the new
implementation is significantly faster, and the code is a lot simpler.
Rewriting AvroStorage also enabled me to implement support for Trevni.
This is the latest version of the patch, complete with test cases and
TrevniStorage. (Test cases for TrevniStorage are still missing).
This addresses bug PIG-3015.
https://issues.apache.org/jira/browse/PIG-3015
Diffs (updated)
-----
.eclipse.templates/.classpath aa9bfd5
build.xml 1f21839
ivy.xml 70e8d50
ivy/libraries.properties bfbbbc0
src/org/apache/pig/builtin/AvroStorage.java PRE-CREATION
src/org/apache/pig/builtin/TrevniStorage.java PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroArrayReader.java PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroBagWrapper.java PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroMapWrapper.java PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroRecordReader.java PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroRecordWriter.java PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroStorageDataConversionUtilities.java
PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroStorageSchemaConversionUtilities.java
PRE-CREATION
src/org/apache/pig/impl/util/avro/AvroTupleWrapper.java PRE-CREATION
test/commit-tests 5081fbc
test/org/apache/pig/builtin/TestAvroStorage.java PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/directory_test.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_ai1_ao2.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_ao2.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_blank_first_args.pig
PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_codec.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_just_ao2.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/namesWithDoubleColons.pig
PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/recursive_tests.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/trevni_to_avro.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/trevni_to_trevni.pig PRE-CREATION
test/org/apache/pig/builtin/avro/createtests.py PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/arrays.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/arraysAsOutputByPig.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordWithRepeatedSubRecords.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/records.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsAsOutputByPig.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsOfArrays.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsOfArraysOfRecords.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsSubSchema.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsSubSchemaNullable.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithDoubleUnderscores.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithEnums.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithFixed.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithMaps.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithMapsOfRecords.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithNullableUnions.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recursiveRecord.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/simpleRecordsTrevni.json
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/arrays.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/arraysAsOutputByPig.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordWithRepeatedSubRecords.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/records.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsAsOutputByPig.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsOfArrays.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsOfArraysOfRecords.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsSubSchema.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsSubSchemaNullable.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithDoubleUnderscores.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc PRE-CREATION
test/unit-tests 7cede06
Diff: https://reviews.apache.org/r/8104/diff/
Testing
-------
Thanks,
Joseph Adler
