Hello,
I'm writing a generic UDAF function that closely resembles SUM() with the
main difference being that it accepts an array datatype parameter and
returns an array datatype.
I've already done this for a GenericUDF successfully. I believe I am having
difficulty coding the proper ObjectInspectors for my parameter & return
objects since I am getting .ClassCastException exceptions for Long ->
LongArray. I am using a hybrid of the GenericUDAFSum.java sample and the
GenericUDAFCollect sample from the Programming Hive book.
My parameter is a fixed length array of longs and the return is the same
length array of longs. As with the SUM function, I do not need to keep the
individual row values that I collect, I can iterate the array, SUM it to
the container and move on to the next row. With this in mind, I think I
can disregard having an internalMergeOI.
Any input is appreciated.
Thanks,
Ross
Here is the exception:
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:441)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:377)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
cast to [Ljava.lang.Object;
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1137)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:199)
... 8 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable
cannot be cast to [Ljava.lang.Object;
at
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector.getListLength(StandardListObjectInspector.java:83)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:418)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:438)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:257)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:204)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:245)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1066)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1118)
... 13 more
Here is the pertinent code:
@Override
public ObjectInspector init(Mode m, ObjectInspector[] parameters)
throws HiveException
{
super.init(m, parameters);
if (m == Mode.PARTIAL1)
{
System.out.println("1 - init() mode: " + m + "
parameter[0]=" + parameters[0].toString());
inputOI = (StandardListObjectInspector) parameters[0];
return
ObjectInspectorFactory.getStandardListObjectInspector(inputOI);
}
else
{
System.out.println("2 - init() mode: " + m + "
parameter[0]=" + parameters[0].toString());
JavaLongObjectInspector doi;
doi =
PrimitiveObjectInspectorFactory.javaLongObjectInspector;
// Set up the list object inspector for the output, and
return it
ListObjectInspector loi;
loi =
ObjectInspectorFactory.getStandardListObjectInspector(doi);
return loi;
// inputOI = (StandardListObjectInspector)
ObjectInspectorUtils.getStandardObjectInspector(parameters[0]);
// return (StandardListObjectInspector)
ObjectInspectorFactory.getStandardListObjectInspector(inputOI);
}
}
static class BitmapAggregationBuffer implements AggregationBuffer {
ArrayList<Object> container;
}
@Override
public Object terminate(AggregationBuffer agg) throws HiveException
{
BitmapAggregationBuffer myAgg = (BitmapAggregationBuffer) agg;
return myAgg.container;
}
--
Ross Levin
Principal Software Engineer
*Simulmedia* | *People Ads Want*
(m) 609.760.5027
670 Broadway, 2nd Floor, New York, NY 10012
*Check out our new data and tools for TV* *Advertising at
**OpenAccess*<http://www.simulmedia.com/OpenAccess/>
*.*
