Even tho this simple thing doesnt work too: grunt> z = foreach data generate time+size;
WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0004 java.lang.ClassCastException: org.apache.pig.data.DataByteArray cannot be cast to java.lang.Float at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.Add.getNext(Add.java:97) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:364) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:291) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:236) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011/4/24 pob <[email protected]> > This one doesnt work... > > > > pom = foreach rows generate myUDF.toTuple($1) AS > (b:bag{t:tuple(domain:chararray,spam:int,size:long,time:float)}); > > 2011-04-24 18:40:15,622 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 1000: Error during parsing. Encountered "" at line 1, column 50 > Was expecting one of: > > > > 2011/4/24 Jacob Perkins <[email protected]> > >> Strange, that looks right to me. What happens if you try the 'AS' >> statement anyhow? >> >> --jacob >> @thedatachef >> >> On Sun, 2011-04-24 at 18:22 +0200, pob wrote: >> > Hello, >> > >> > pom = foreach rows generate myUDF.toTuple($1); -- reading data >> > describe pom >> > pom: {y: {t: (domain: chararray,spam: int,size: long,time: float)}} >> > >> > data = foreach pom generate flatten($0); >> > grunt> describe data; >> > data: {y::domain: chararray,y::spam: int,y::size: long,y::time: float} >> > >> > >> > I thing they are casted fine, right? >> > >> > UDF is python one with decorator >> > @outputSchema("y:bag{t:tuple(domain:chararray, spam:int, size:long, >> > time:float)}") >> > >> > Thanks >> > >> > >> > >> > 2011/4/24 Jacob Perkins <[email protected]> >> > >> > > You're getting a 'ClassCastException' because the contents of the bags >> > > are DataByteArray and not long (or cannot be cast to long). I suspect >> > > that you're generating the contents of the bag in some way from a UDF, >> > > no? >> > > >> > > You need to either declare the output schema explicitly in the UDF or >> > > just use the 'AS' statement. For example, say you have a UDF that sums >> > > two numbers: >> > > >> > > data = LOAD 'foobar' AS (int:a, int:b); >> > > summed = FOREACH data GENERATE MyFancySummingUDF(a,b) AS (sum:int); >> > > DUMP summed; >> > > >> > > --jacob >> > > @thedatachef >> > > >> > > On Sun, 2011-04-24 at 18:02 +0200, pob wrote: >> > > > x = foreach g2 generate group, data.(size); >> > > > dump x; >> > > > >> > > > ((drm,0),{(464868)}) >> > > > ((drm,1),{(464868)}) >> > > > ((snezz,0),{(8073),(8073)}) >> > > > >> > > > but: >> > > > x = foreach g2 generate group, SUM(data.size); >> > > > >> > > > >> > > > >> > > > >> > > > 2011-04-24 18:02:18,910 [Thread-793] WARN >> > > > org.apache.hadoop.mapred.LocalJobRunner - job_local_0038 >> > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2106: >> Error >> > > > while computing sum in Initial >> > > > at org.apache.pig.builtin.LongSum$Initial.exec(LongSum.java:87) >> > > > at org.apache.pig.builtin.LongSum$Initial.exec(LongSum.java:65) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:229) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:273) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:343) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:291) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:276) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:236) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231) >> > > > at >> > > > >> > > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) >> > > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) >> > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) >> > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) >> > > > at >> > > >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) >> > > > Caused by: java.lang.ClassCastException: >> > > org.apache.pig.data.DataByteArray >> > > > cannot be cast to java.lang.Long >> > > > at org.apache.pig.builtin.LongSum$Initial.exec(LongSum.java:79) >> > > > ... 14 more >> > > > 2011-04-24 18:02:19,213 [main] INFO >> > > > >> > > >> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > > > - HadoopJobId: job_local_0038 >> > > > 2011-04-24 18:02:19,213 [main] INFO >> > > > >> > > >> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > > > - 0% complete >> > > > 2011-04-24 18:02:24,215 [main] INFO >> > > > >> > > >> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > > > - job job_local_0038 has failed! Stop running all dependent jobs >> > > > 2011-04-24 18:02:24,216 [main] INFO >> > > > >> > > >> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > > > - 100% complete >> > > > 2011-04-24 18:02:24,216 [main] ERROR >> > > > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) >> failed! >> > > > 2011-04-24 18:02:24,216 [main] INFO >> > > org.apache.pig.tools.pigstats.PigStats >> > > > - Detected Local mode. Stats reported below may be incomplete >> > > > 2011-04-24 18:02:24,216 [main] INFO >> > > org.apache.pig.tools.pigstats.PigStats >> > > > - Script Statistics: >> > > > >> > > > >> > > > >> > > > >> > > > Pig Stack Trace >> > > > --------------- >> > > > ERROR 1066: Unable to open iterator for alias x >> > > > >> > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: >> Unable to >> > > > open iterator for alias x >> > > > at org.apache.pig.PigServer.openIterator(PigServer.java:754) >> > > > at >> > > > >> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612) >> > > > at >> > > > >> > > >> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303) >> > > > at >> > > > >> > > >> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165) >> > > > at >> > > > >> > > >> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141) >> > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76) >> > > > at org.apache.pig.Main.run(Main.java:465) >> > > > at org.apache.pig.Main.main(Main.java:107) >> > > > Caused by: java.io.IOException: Job terminated with anomalous status >> > > FAILED >> > > > at org.apache.pig.PigServer.openIterator(PigServer.java:744) >> > > > ... 7 more >> > > >> > > >> > > >> >> >> >
