Hmm, in that case it allows me to run it, but dies and gives me this error
(from log)
Pig Stack Trace
---------------
ERROR 1066: Unable to open iterator for alias C
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias C
at org.apache.pig.PigServer.openIterator(PigServer.java:754)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:744)
... 7 more
================================================================================
and here is what grunt gave
java.lang.RuntimeException: Unexpected data type -1 found in stream.
at
org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:478)
at
org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541)
at
org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357)
at
org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541)
at
org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357)
at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:57)
at
org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:900)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
not sure what this -1 business is...
2011/1/14 Dmitriy Ryaboy <[email protected]>
> Much as it pains me -- what happens if you ungenerify (just use Map instead
> of Map<String, Integer>)?
>
> On Fri, Jan 14, 2011 at 3:06 PM, Jonathan Coveney <[email protected]
> >wrote:
>
> > I did a search online, and while someone had the same error, I don't
> think
> > it was related. From the error log, I see this...
> >
> > Caused by: java.lang.RuntimeException: Final function of
> > squeal.map.makecountalgq is not of the expected type.
> > at org.apache.pig.EvalFunc.<init>(EvalFunc.java:146)
> > at squeal.map.makecountalgq.<init>(makecountalgq.java:32)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at java.lang.Class.newInstance0(Class.java:355)
> > at java.lang.Class.newInstance(Class.java:308)
> > at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:472)
> >
> > But here is my udf (I've truncated a bit, but it works if it isn't
> > algebraic)
> >
> > public class makecountalgq extends EvalFunc<Map<String,Integer>>
> implements
> > Algebraic, Accumulator<Map<String,Integer>> {
> > static TupleFactory mTupleFactory = TupleFactory.getInstance();
> > static BagFactory mBagFactory = BagFactory.getInstance();
> > private MapWrapper ourWrap = new MapWrapper();
> >
> > public String getInitial() { return Initial.class.getName(); }
> >
> > public String getIntermed() { return Intermed.class.getName(); }
> >
> > public String getFinal() { return Final.class.getName(); }
> >
> > //An initial tuple, crates partial results
> > public static class Initial extends EvalFunc<Tuple> {
> > public Tuple exec(Tuple input) throws IOException {
> > MapWrapper aWrap = new MapWrapper(new
> > HashMap<String,MutableInt>());
> > DataBag bag = (DataBag)input.get(0);
> > aWrap.addSBag(bag);
> > return mTupleFactory.newTuple(aWrap);
> > }
> > }
> > //Called on the result of the Initial, or on another intermediate
> > public static class Intermed extends EvalFunc<Tuple> {
> > public Tuple exec(Tuple input) throws IOException {
> > return
> > mTupleFactory.newTuple(mergeWrappers((DataBag)input.get(0)));
> > }
> > }
> > //Can take the tuple that initial or intermed creates
> > public static class Final extends EvalFunc<Map<String,Integer>> {
> > public Map<String,Integer> exec(Tuple input) throws
> > IOException {
> > return
> > mergeWrappers((DataBag)input.get(0)).convertMap();
> > }
> > }
> >
> > //takes a bag of MapWrappers and merges them
> > public static MapWrapper mergeWrappers(DataBag bag) throws
> > IOException {
> > Iterator<Tuple> bagIt = bag.iterator();
> > MapWrapper aWrap = (MapWrapper)bagIt.next().get(0);
> > while (bagIt.hasNext())
> > aWrap.merge((MapWrapper)bagIt.next().get(0));
> > return aWrap;
> > }
> > }
> >
> > Any idea why I would get this error? It compiles fine, and if I even just
> > take out Algebraic, I can use the UDF (it just uses the accumulator). I
> > don't see how the type is off...
> >
> > Any ideas? (in case the context matters: this is a script that takes a
> list
> > of (a, b:chararray), and gives you, for a given key, a map from b-># of
> > times b occured.
> >
> > Here is how I invoke it:
> >
> > register squeal.jar;
> > A = LOAD 'test.txt' AS (a:chararray, b:chararray);
> > B = GROUP A BY a;
> > C = FOREACH B GENERATE group, squeal.map.makecountalgq(A.b) as mapz;
> >
> > And I get the error.
> >
> > Thanks in advance. I guess I could content myself with an Accumulator but
> > the Algebraic solution is so close... a MapWrapper is a class I created
> > that
> > lets me pass around Maps and whatnot.
> >
>
