Hi Dan, Glad to hear that it worked. I totally agree that AvroStorage can be improved. In fact, it was written for Pig 0.7, so it can be written much nicer now.
Only concern that I have is backward compatibility. That is, if I change syntax (I wanted so badly while working on AvroStorage recently), it will break backward compatibility. What I have been thinking is to rewrite AvroStorage in core Pig like HBaseStorage. For backward compatibility, we may keep the old version in Piggybank for a while and eventually retire it. I am wondering what other people think. Please let me know if it is not a good idea to move AvroStorage to core Pig from Piggybank. Thanks, Cheolsoo On Tue, Aug 21, 2012 at 5:47 PM, Danfeng Li <[email protected]> wrote: > Thanks, Cheolsoo. That solve my problems. > > It will be nice if pig can do this automatically when there are multiple > avrostorage in the code. Otherwise, we have to manually track the numbers. > > Dan > > -----Original Message----- > From: Cheolsoo Park [mailto:[email protected]] > Sent: Tuesday, August 21, 2012 5:06 PM > To: [email protected] > Subject: Re: runtime exception when load and store multiple files using > avro in pig > > Hi Danfeng, > > The "long" is from the 1st AvroStorage store in your script. The > AvroStorage has very funny syntax regarding multiple stores. To apply > different avro schemas to multiple stores, you have to specify their > "index" as follows: > > set1 = load 'input1.txt' using PigStorage('|') as ( ... ); *store set1 > into 'set1' using > org.apache.pig.piggybank.storage.avro.AvroStorage('index', '1');* > > set2 = load 'input2.txt' using PigStorage('|') as ( .. ); *store set2 into > 'set2' using org.apache.pig.piggybank.storage.avro.AvroStorage('index', > '2');* > > As can be seen, I added the 'index' parameters. > > What AvroStorage does is to construct the following string in the frontend: > > "1#<1st avro schema>,2#<2nd avro schema>" > > and pass it to backend via UdfContext. Now in backend, tasks parse this > string to get output schema for each store. > > Thanks, > Cheolsoo > > On Tue, Aug 21, 2012 at 4:38 PM, Danfeng Li <[email protected]> > wrote: > > > I run into this strange problem when try to load multiple text > > formatted files and convert them into avro format using pig. However, > > if I read and convert one file at a time in separated runs, everything > > is fine. The error message is following > > > > 2012-08-21 19:15:32,964 [main] ERROR > > org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to > > recreate exception from backed error: > > org.apache.avro.file.DataFileWriter$AppendWriteException: > > java.lang.RuntimeException: Datum 1980-01-01 00:00:00.000 is not in > > union ["null","long"] > > at > > org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263) > > at > > > org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49) > > at > > > org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:612) > > at > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) > > at > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) > > at > > > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531) > > at > > > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > > at > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGeneri > > cMapB > > > > my code is > > set1 = load '$input_dir/set1.txt' using PigStorage('|') as ( > > id:long, > > f1:long, > > f2:chararray, > > f3:float, > > f4:float, > > f5:float, > > f6:float, > > f7:float, > > f8:float, > > f9:float, > > f10:float, > > f11:float, > > f12:float); > > store set1 into '$output_dir/set1.avro' > > using org.apache.pig.piggybank.storage.avro.AvroStorage(); > > > > set2 = load '$input_dir/set2.txt' using PigStorage('|') as ( > > id : int, > > date : chararray); > > store set2 into '$output_dir/set2.avro' > > using org.apache.pig.piggybank.storage.avro.AvroStorage(); > > > > The first file is converted fine, but the 2nd one is failed. The error > > is coming from the 2nd field in the 2nd file, but the strange thing is > > that I don't even have "long" in my schema while the error message is > > showing ["null","long"]. > > > > I use pig 0.10.0 and avro-1.7.1.jar. > > > > I wonder if this is a bug or I missed something. > > > > Thanks. > > Dan > > > > Here's set1.txt > > > > 827352|740214|Long|26|0.08731795012183759|1661335.541733333|0|0|0.0010 > > 827352|740214|Long|26|57865808239878|0.001059541098077884|0.0010595410 > > 827352|740214|Long|26|98077821|0.0514156486228232|0.001043980181757539 > > > > 827353|740214|Short|12|-0.05967910581502997|-1135471.22271|0|0|-0.0011 > > 827353|740214|Short|12|85620143839061|-0.001187497751909232|-0.0011874 > > 827353|740214|Short|12|97751909183|-0.0747641932858414|-0.000130744900 > > 827353|740214|Short|12|2148424 > > > > 827354|740214|Total|38|0.02763884430680765|19026277.40819863|0|0|-0.00 > > 827354|740214|Total|38|01277543355991829|-0.0001279566538313473|-0.000 > > 827354|740214|Total|38|1279566538313626|-0.02334854466301821|0.0009132 > > 827354|740214|Total|38|352815426966 > > > > 827193|739576|Long|26|0.08731795012183759|1661335.541733333|0|0|0.0010 > > 827193|739576|Long|26|57865808239878|0.001059541098077884|0.0010595410 > > 827193|739576|Long|26|98077821|0.0514156486228232|0.001043980181757539 > > > > 827194|739576|Short|12|-0.05967910581502997|-1135471.22271|0|0|-0.0011 > > 827194|739576|Short|12|85620143839061|-0.001187497751909232|-0.0011874 > > 827194|739576|Short|12|97751909183|-0.0747641932858414|-0.000130744900 > > 827194|739576|Short|12|2148424 > > > > 827195|739576|Total|38|0.02763884430680765|19026277.40819863|0|0|-0.00 > > 827195|739576|Total|38|01277543355991829|-0.0001279566538313473|-0.000 > > 827195|739576|Total|38|1279566538313626|-0.02334854466301821|0.0009132 > > 827195|739576|Total|38|352815426966 > > > > 827355|740215|Long|51|1.776868012839072|113652088.7063555|0|0|0.019525 > > 827355|740215|Long|51|47658695701|0.0195703176808393|0.019570317680839 > > 827355|740215|Long|51|28|1.164818333642054|0 > > > > 827356|740215|Short|34|-2.360589090333165|-150988074.9471841|0|0|-0.00 > > 827356|740215|Short|34|868330219442376|-0.008616238065508337|-0.008616 > > 827356|740215|Short|34|238065508375|-0.5943698959308671|-0.02690679230 > > 827356|740215|Short|34|502523 > > > > 827357|740215|Total|85|-0.5837210774940929|63962032.00527128|0|0|0.010 > > 827357|740215|Total|85|84217439253325|0.01095407961533095|0.0109540796 > > 827357|740215|Total|85|153309|0.5704484377111866|-0.02690679230502523 > > > > 827202|739590|Long|53|1.777568428360522|113696888.7063555|0|0|0.019525 > > 827202|739590|Long|53|47658695701|0.0195703176808393|0.019570317680839 > > 827202|739590|Long|53|28|1.156653489849146|0 > > > > Here's the set2.txt > > 1|1980-01-01 00:00:00.000 > > 2|1980-01-02 00:00:00.000 > > 3|1980-01-03 00:00:00.000 > > 4|1980-01-04 00:00:00.000 > > 5|1980-01-07 00:00:00.000 > > 6|1980-01-08 00:00:00.000 > > 7|1980-01-09 00:00:00.000 > > 8|1980-01-10 00:00:00.000 > > 9|1980-01-11 00:00:00.000 > > 10|1980-01-14 00:00:00.000 > > > > >
