[
https://issues.apache.org/jira/browse/PIG-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986551#action_12986551
]
Felix Gao commented on PIG-1748:
--------------------------------
I noticed the avro loader does not support file globbing.
log_load = LOAD '/user/felix/avro/access_log.test.avro' USING
org.apache.pig.piggybank.storage.avro.AvroStorage(); <--- works fine
but
log_load = LOAD '/user/felix/avro/*.avro' USING
org.apache.pig.piggybank.storage.avro.AvroStorage();
ERROR 1018: Problem determining schema during load
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during
parsing. Problem determining schema during load
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1342)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
at org.apache.pig.Main.main(Main.java:414)
Caused by: org.apache.pig.impl.logicalLayer.parser.ParseException: Problem
determining schema during load
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:752)
at
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
... 8 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1018:
Problem determining schema during load
at org.apache.pig.impl.logicalLayer.LOLoad.getSchema(LOLoad.java:156)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:750)
... 10 more
Caused by: java.io.FileNotFoundException: File does not exist:
/user/felix/avro/*.avro
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1586)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1577)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:428)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:185)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:431)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:181)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:133)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:108)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:233)
at org.apache.pig.impl.logicalLayer.LOLoad.determineSchema(LOLoad.java:169)
at org.apache.pig.impl.logicalLayer.LOLoad.getSchema(LOLoad.java:150)
... 11 more
> Add load/store function AvroStorage for avro data
> -------------------------------------------------
>
> Key: PIG-1748
> URL: https://issues.apache.org/jira/browse/PIG-1748
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: lin guo
> Assignee: Jakob Homan
> Attachments: avro_storage.patch, avro_test_files.tar.gz,
> PIG-1748-2.patch
>
>
> We want to use Pig to process arbitrary Avro data and store results as Avro
> files. AvroStorage() extends two PigFuncs: LoadFunc and StoreFunc.
> Due to discrepancies of Avro and Pig data models, AvroStorage has:
> 1. Limited support for "record": we do not support recursively defined record
> because the number of fields in such records is data dependent.
> 2. Limited support for "union": we only accept nullable union like ["null",
> "some-type"].
> For simplicity, we also make the following assumptions:
> If the input directory is a leaf directory, then we assume Avro data files in
> it have the same schema;
> If the input directory contains sub-directories, then we assume Avro data
> files in all sub-directories have the same schema.
> AvroStorage takes no input parameters when used as a LoadFunc (except for
> "debug [debug-level]").
> Users can provide parameters to AvroStorage when used as a StoreFunc. If they
> don't, Avro schema of output data is derived from its
> Pig schema.
> Detailed documentation can be found in
> http://snaprojects.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
