Hi Zheng,
I made the change the FunctionRegistry.java file and rebuild the Hive code
base. After re-running the UDF, I got the following exceptions for illegal
Hex characters in escape (%) pattern. So in my UDF function, I have try and
exception catcher, shouldn't the value return null on the exception?
Thanks,
-ray
java.lang.RuntimeException: Unable to execute method public
org.apache.hadoop.io.Text
com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
com.sharethis.norm.urldec...@6f9ec4ed of class com.test.norm.URLDecode
with arguments
{http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
of size 1
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
execute method public org.apache.hadoop.io.Text
com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
com.test.norm.urldec...@6f9ec4ed of class com.test.norm.URLDecode with
arguments
{http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
of size 1
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
at
org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
at
org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
... 11 more
Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
characters in escape (%) pattern - For input string: "&I"
at java.net.URLDecoder.decode(URLDecoder.java:173)
at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
... 15 more
java.lang.RuntimeException: Unable to execute method public
org.apache.hadoop.io.Text
com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
com.sharethis.norm.urldec...@6a5eb489 of class com.test.norm.URLDecode
with arguments
{http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
of size 1
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
execute method public org.apache.hadoop.io.Text
com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
com.sharethis.norm.urldec...@6a5eb489 of class com.test.norm.URLDecode
with arguments
{http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
of size 1
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
at
org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
at
org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
... 11 more
Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
characters in escape (%) pattern - For input string: "&I"
at java.net.URLDecoder.decode(URLDecoder.java:173)
at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
... 15 more
On Thu, Aug 20, 2009 at 10:48 PM, Zheng Shao <[email protected]> wrote:
>
> http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034
>
> Can you change the following several lines?
>
> e.printStackTrace();
> throw new HiveException("Unable to execute method " + m + " "
> + " on object " + thisObjectString
> + " with arguments " + argumentString.toString()
> + ":" + e.getMessage());
>
> to:
>
> throw new HiveException("Unable to execute method " + m + " "
> + " on object " + thisObjectString
> + " with arguments " + argumentString.toString(), e);
>
>
> Some exception were thrown but unfortunately e.getMessage() returns
> "null". With this change, we will know what exactly happened.
>
> Zheng
>
> On Thu, Aug 20, 2009 at 4:29 PM, Ray Duong<[email protected]> wrote:
> > Hi,
> >
> > I built a UDF function to decode the URLs from the log files. When I run
> > the script in Hive, i get the following error message. Is there
> something I
> > should be catching beside the NULL values?
> >
> >
> > Thanks,
> > -ray
> >
> > java.lang.RuntimeException: Unable to execute method public
> > org.apache.hadoop.io.Text
> > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
> > com.test.norm.urldec...@27b03c1a of class com.test.norm.URLDecode with
> > arguments
> >
> {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
> > of size 1:null
> >
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> > at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> > execute method public org.apache.hadoop.io.Text
> > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
> > com.test.norm.urldec...@27b03c1a of class com.test.norm.URLDecode with
> > arguments
> >
> {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
> > of size 1:null
> >
> > at
> >
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
> > at
> >
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
> > at
> >
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
> >
> > at
> >
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
> > at
> >
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
> >
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
> > at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
> >
> > ... 3 more
> >
> >
> > Below is the UDF class;
> > package com.test.norm;
> >
> > import org.apache.hadoop.hive.ql.exec.UDF;
> > import org.apache.hadoop.io.Text;
> > import org.apache.log4j.Logger;
> > import java.io.UnsupportedEncodingException;
> > import java.net.URLDecoder;
> >
> >
> > public final class URLDecode extends UDF {
> >
> >
> >
> > private static final Logger LOG =
> Logger.getLogger(URLDecode.class);
> >
> >
> > public Text evaluate(final Text s) {
> > if (s == null) {
> > LOG.error("input is null");
> > return null;
> > }
> >
> >
> > String url_orig = s.toString();
> >
> > if (url_orig == null || url_orig.isEmpty()) {
> > LOG.error("orig url is null");
> > return null;
> > }
> >
> > String url_norm = null;
> >
> > try {
> > url_norm = URLDecoder.decode(url_orig, "UTF-8");
> > // url_norm = URLDecoder.decode(url_norm, "UTF-8");
> > } catch (Exception exception) {
> > LOG.error("wrong url code :::" + exception.getMessage());
> > return null;
> > }
> > if (url_norm == null) {
> > LOG.error("cant decode from " + s);
> > return null;
> > }
> > Text to_value = new Text();
> > to_value.set(url_norm);
> > return to_value;
> > }
> > }
> >
> >
> >
> >
>
>
>
> --
> Yours,
> Zheng
>
