[
https://issues.apache.org/jira/browse/PIG-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657784#comment-13657784
]
Viraj Bhat commented on PIG-3322:
---------------------------------
It seems that the schema specified during load time is stored in
"outputAvroSchema" but is not used when reading the underlying data. It will be
used when writing out the data.
PIG-3321 will enable to use this schema when reading the data but will need to
investigate if it fixes the above problem.
> AVRO: AvroStorage give NPE on reading file with union as top level schema
> -------------------------------------------------------------------------
>
> Key: PIG-3322
> URL: https://issues.apache.org/jira/browse/PIG-3322
> Project: Pig
> Issue Type: Bug
> Components: piggybank
> Affects Versions: 0.11.2
> Reporter: Egil Sorensen
> Assignee: Viraj Bhat
> Labels: patch
> Fix For: 0.12, 0.11.2
>
>
> I am getting NPE when loading a file with AvroStorage a file that has schema
> like:
> {code}
> ["null",{"type":"record","name":"TUPLE_0","fields":[{"name":"name","type":["null","string"],"doc":"autogenerated
> from Pig Field
> Schema"},{"name":"age","type":["null","int"],"doc":"autogenerated from Pig
> Field Schema"},{"name":"gpa","type":["null","double"],"doc":"autogenerated
> from Pig Field Schema"}]}]
> {code}
> E.g. see the e2e style test, which fails on this:
> {code}
> {
> 'num' => 4,
> # storing file with Pig type tuple relying on
> conversion to record
> # loading using stored schemas
> 'notmq' => 1,
> 'pig' => q\
> a = load ':INPATH:/singlefile/studentcomplextab10k' using PigStorage() as
> (m:[], t:(name:chararray, age:int, gpa:double), b:{t:(name:chararray,
> age:int, gpa:double)});
> b = foreach a generate t;
> describe b;
> store b into ':OUTPATH:.intermediate' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
> exec;
> -- Read back what was stored with Avro
> u = load ':OUTPATH:.intermediate' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
> describe u;
> store u into ':OUTPATH:';
> \,
> 'verify_pig_script' => q\
> a = load ':INPATH:/singlefile/studentcomplextab10k' using PigStorage() as
> (m:[], t:(name:chararray, age:int, gpa:double), b:{t:(name:chararray,
> age:int, gpa:double)});
> b = foreach a generate t;
> describe b;
> store b into ':OUTPATH:';
> \,
> },
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
