Hi Robin,
Thanks for the response.
The point mentioned by you is one of the many other serious issues in the
code.
I have fixed all of them and it is pretty much in a good shape presently
after some testing.
You can find it at:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Let me know if you face any other issues!
Thanks,
Abhishek
On Wed, Feb 13, 2013 at 7:02 PM, Robin Morris <r...@baynote.com> wrote:
> Hi,
>
> You might well have found the error yourself by now, but if not, the
> problem is that you missed the "static" keyword off the declaration of
> TopNPercentEvaluator. Line 18 of your code should read
>
> public static class TopNPercentEvaluator implements UDAFEvaluator {
>
> then this error goes away.
>
> A new error appears, however, because there's a bug in your terminate()
> function that reads off the end of the array.
>
> Robin
>
>
> From: Abhishek Bhattacharya <abhat...@fiu.edu>
> Reply-To: "user@hive.apache.org" <user@hive.apache.org>
> Date: Tuesday, February 12, 2013 10:29 AM
> To: "user@hive.apache.org" <user@hive.apache.org>
> Subject: Fwd: Help to solve UDAF errors!
>
> Hi,
>
> I looked through the syslogs and found the following exceptions.
> Can anyone help me to figure out the point of error?
>
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
> Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException:
> com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:62)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
> ... 22 more
> Caused by: java.lang.NoSuchMethodException:
> com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
> at java.lang.Class.getConstructor0(Class.java:2706)
> at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> ... 37 more
> 2013-02-12 07:21:14,049 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
> for the task
>
> ------------------------------
> Thanks,
> Abhishek
>
>
>
> ---------- Forwarded message ----------
> From: Abhishek Bhattacharya <abhat...@fiu.edu>
> Date: Tue, Feb 12, 2013 at 12:48 PM
> Subject: Re: Help to solve UDAF errors!
> To: user@hive.apache.org
>
>
> Hi Mark,
>
> Thanks for the response!
> The UDAFPercentile.java have two terminate() methods since it is handling
> two different input types by the two inner classes: PercentileLongEvaluator
> and PercentileLongArrayEvaluator.
> I am handling only a single input type of double from one table column to
> the iterate() method and wish to return an ArrayList<DoubleWritable> from
> the terminate() method.
> What is wrong in my class?
> Moreover, is there any way for UDF/UDAF/UDTF which can process all the
> rows of the table and output only a subset of the total rows based on some
> aggregation function of one column attribute i.e., similar to my case of
> computing the top-n-percent of a column attribute and output the entire set
> of filtered rows with all other columns from the table?
>
> Thanks,
> Abhishek
>
>
>
> On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover <grover.markgro...@gmail.com
> > wrote:
>
>> Hi Abhishek,
>> The code looks incomplete.
>>
>> See the comment at
>> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22
>> Those are all the methods your UDAF class needs to implement but you seem
>> to be missing them.
>>
>> Mark
>>
>> On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya
>> <abhat...@fiu.edu>wrote:
>>
>>> Thanks for the response.
>>> The link to the code is:
>>> https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
>>> Please let me know to fix it!
>>>
>>> Thanks,
>>> Abhishek
>>>
>>>
>>>
>>> On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <grover.markgro...@gmail.com
>>> > wrote:
>>>
>>>> Abhishek,
>>>> The code doesn't seem to be complete.
>>>>
>>>> Look at
>>>> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor
>>>> reference. It has two terminate()'s - one for UDAF and one for the
>>>> Evaluator.
>>>>
>>>> Do you mind posting your complete code on github somewhere so it's
>>>> easier to analyze?
>>>>
>>>> Mark
>>>>
>>>> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <abhat...@fiu.edu
>>>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have implemented a simple UDAF for top-n-percent as follows:
>>>>> import java.util.ArrayList;
>>>>> import java.util.Collections;
>>>>>
>>>>> import org.apache.hadoop.hive.ql.exec.UDAF;
>>>>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>>>>
>>>>> public class UDAFTopNPercent extends UDAF{
>>>>>
>>>>> public static class Result {
>>>>> ArrayList<Double> list;
>>>>> double min;
>>>>> }
>>>>>
>>>>> public class TopNPercentEvaluator implements UDAFEvaluator {
>>>>>
>>>>> private Result res;
>>>>> private int rowIndex;
>>>>> private int percent;
>>>>>
>>>>> public TopNPercentEvaluator() {
>>>>> super();
>>>>> res = new Result();
>>>>> init();
>>>>> rowIndex = 0;
>>>>> }
>>>>> @Override
>>>>> public void init() {
>>>>> res.list = new ArrayList<Double>();
>>>>> res.min = Double.MAX_VALUE;
>>>>> }
>>>>>
>>>>> public boolean iterate(Double rowVal, int pct) {
>>>>> ArrayList<Double> resList = res.list;
>>>>> rowIndex++;
>>>>> resList.add(rowVal);
>>>>> percent = pct;
>>>>> return true;
>>>>> }
>>>>>
>>>>> public ArrayList<Double> terminatePartial() {
>>>>> ArrayList<Double> resList = res.list;
>>>>> Collections.sort(resList);
>>>>> return resList;
>>>>> }
>>>>>
>>>>> public boolean merge(ArrayList<Double> otherList) {
>>>>> ArrayList<Double> resList = res.list;
>>>>> resList.addAll(otherList);
>>>>> return true;
>>>>> }
>>>>>
>>>>> public ArrayList<Double> terminate() {
>>>>> ArrayList<Double> resList = res.list;
>>>>> double num_rows = (double)percent/100.0*rowIndex;
>>>>> Collections.sort(resList);
>>>>> int lastIdx = resList.size()- (int) num_rows;
>>>>> if(lastIdx <= 0) {
>>>>> return resList;
>>>>> }
>>>>> for(int i=0; i<lastIdx; i++) {
>>>>> resList.remove(i);
>>>>> }
>>>>> return resList;
>>>>> }
>>>>> }
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> public static void main(String[] args) {
>>>>> // TODO Auto-generated method stub
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> But throws some error such as first few lines are:
>>>>> FAILED: Hive Internal Error:
>>>>> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>>>>> cannot be cast to
>>>>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
>>>>> java.lang.ClassCastException:
>>>>> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>>>>> cannot be cast to
>>>>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
>>>>> at
>>>>> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)
>>>>>
>>>>> Please help me to debug this!
>>>>> Is it throwing from returning ArrayList<Double> in terminate()?
>>>>> How should I return a List from UDAF?
>>>>>
>>>>> Thanks,
>>>>> Abhishek
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Abhishek Bhattacharya
>>> PhD Computer Science
>>> School of Computing and Information Sciences
>>> Florida International University
>>>
>>
>>
>
>
> --
> Thanks and Regards,
>
> Abhishek Bhattacharya
> PhD Computer Science
> School of Computing and Information Sciences
> Florida International University
>
>
>
> --
> Thanks and Regards,
>
> Abhishek Bhattacharya
> PhD Computer Science
> School of Computing and Information Sciences
> Florida International University
>
--
Thanks and Regards,
Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University
