Hi Zheng, I was able resolved the problem by putting a isEmpty() along with the null string condition checker. This seems to resolve the issue.
Thanks, -ray On Tue, Aug 25, 2009 at 1:13 AM, Zheng Shao <[email protected]> wrote: > Not sure if you figured this out or not, but the stack trace clearly shows > there is a problem at the URLDecode.java:25. > Maybe you didn't redeploy the jar after changing the code. > > > Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex > characters in escape (%) pattern - For input string: "&I" > > at java.net.URLDecoder.decode(URLDecoder.java:173) > at *com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)* > ... 15 more > > Zheng > > > > On Fri, Aug 21, 2009 at 7:48 AM, Ray Duong<[email protected]> wrote: > > Hi Zheng, > > > > I made the change the FunctionRegistry.java file and rebuild the Hive > code > > base. After re-running the UDF, I got the following exceptions for > illegal > > Hex characters in escape (%) pattern. So in my UDF function, I have try > and > > exception catcher, shouldn't the value return null on the exception? > > > > Thanks, > > -ray > > > > java.lang.RuntimeException: Unable to execute method public > > org.apache.hadoop.io.Text > > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object > > com.sharethis.norm.urldec...@6f9ec4ed of class com.test.norm.URLDecode > with > > arguments > > {http%3A%2F%2Fwww.swapalease.com > %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text} > > of size 1 > > > > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182) > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > > at org.apache.hadoop.mapred.Child.main(Child.java:158) > > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > > execute method public org.apache.hadoop.io.Text > > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object > > com.test.norm.urldec...@6f9ec4ed of class com.test.norm.URLDecode with > > arguments > > {http%3A%2F%2Fwww.swapalease.com > %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text} > > of size 1 > > > > at > > > org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478) > > at > > > org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131) > > at > > > org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99) > > > > at > > > org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72) > > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486) > > at > > > org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42) > > > > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486) > > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297) > > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165) > > > > ... 3 more > > Caused by: java.lang.reflect.InvocationTargetException > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454) > > ... 11 more > > Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex > > characters in escape (%) pattern - For input string: "&I" > > > > at java.net.URLDecoder.decode(URLDecoder.java:173) > > at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25) > > > > > > ... 15 more > > > > java.lang.RuntimeException: Unable to execute method public > > org.apache.hadoop.io.Text > > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object > > com.sharethis.norm.urldec...@6a5eb489 of class com.test.norm.URLDecode > with > > arguments > > {http%3A%2F%2Fwww.swapalease.com > %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text} > > of size 1 > > > > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182) > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > > at org.apache.hadoop.mapred.Child.main(Child.java:158) > > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > > execute method public org.apache.hadoop.io.Text > > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object > > com.sharethis.norm.urldec...@6a5eb489 of class com.test.norm.URLDecode > with > > arguments > > {http%3A%2F%2Fwww.swapalease.com > %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text} > > of size 1 > > > > at > > > org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478) > > at > > > org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131) > > at > > > org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99) > > > > at > > > org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72) > > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486) > > at > > > org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42) > > > > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486) > > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297) > > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165) > > > > ... 3 more > > Caused by: java.lang.reflect.InvocationTargetException > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454) > > ... 11 more > > Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex > > characters in escape (%) pattern - For input string: "&I" > > > > at java.net.URLDecoder.decode(URLDecoder.java:173) > > at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25) > > ... 15 more > > > > > > > > On Thu, Aug 20, 2009 at 10:48 PM, Zheng Shao <[email protected]> wrote: > >> > >> > >> > http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034 > >> > >> Can you change the following several lines? > >> > >> e.printStackTrace(); > >> throw new HiveException("Unable to execute method " + m + " " > >> + " on object " + thisObjectString > >> + " with arguments " + argumentString.toString() > >> + ":" + e.getMessage()); > >> > >> to: > >> > >> throw new HiveException("Unable to execute method " + m + " " > >> + " on object " + thisObjectString > >> + " with arguments " + argumentString.toString(), e); > >> > >> > >> Some exception were thrown but unfortunately e.getMessage() returns > >> "null". With this change, we will know what exactly happened. > >> > >> Zheng > >> > >> On Thu, Aug 20, 2009 at 4:29 PM, Ray Duong<[email protected]> wrote: > >> > Hi, > >> > > >> > I built a UDF function to decode the URLs from the log files. When I > >> > run > >> > the script in Hive, i get the following error message. Is there > >> > something I > >> > should be catching beside the NULL values? > >> > > >> > > >> > Thanks, > >> > -ray > >> > > >> > java.lang.RuntimeException: Unable to execute method public > >> > org.apache.hadoop.io.Text > >> > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object > >> > com.test.norm.urldec...@27b03c1a of class com.test.norm.URLDecode > with > >> > arguments > >> > > >> > > {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text} > >> > of size 1:null > >> > > >> > at > >> > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182) > >> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > >> > at org.apache.hadoop.mapred.Child.main(Child.java:158) > >> > > >> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > >> > execute method public org.apache.hadoop.io.Text > >> > com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object > >> > com.test.norm.urldec...@27b03c1a of class com.test.norm.URLDecode > with > >> > arguments > >> > > >> > > {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text} > >> > of size 1:null > >> > > >> > at > >> > > >> > > org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478) > >> > at > >> > > >> > > org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131) > >> > at > >> > > >> > > org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99) > >> > > >> > at > >> > > >> > > org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72) > >> > at > >> > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486) > >> > at > >> > > >> > > org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42) > >> > > >> > at > >> > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486) > >> > at > >> > > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297) > >> > at > >> > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165) > >> > > >> > ... 3 more > >> > > >> > > >> > Below is the UDF class; > >> > package com.test.norm; > >> > > >> > import org.apache.hadoop.hive.ql.exec.UDF; > >> > import org.apache.hadoop.io.Text; > >> > import org.apache.log4j.Logger; > >> > import java.io.UnsupportedEncodingException; > >> > import java.net.URLDecoder; > >> > > >> > > >> > public final class URLDecode extends UDF { > >> > > >> > > >> > > >> > private static final Logger LOG = > >> > Logger.getLogger(URLDecode.class); > >> > > >> > > >> > public Text evaluate(final Text s) { > >> > if (s == null) { > >> > LOG.error("input is null"); > >> > return null; > >> > } > >> > > >> > > >> > String url_orig = s.toString(); > >> > > >> > if (url_orig == null || url_orig.isEmpty()) { > >> > LOG.error("orig url is null"); > >> > return null; > >> > } > >> > > >> > String url_norm = null; > >> > > >> > try { > >> > url_norm = URLDecoder.decode(url_orig, "UTF-8"); > >> > // url_norm = URLDecoder.decode(url_norm, "UTF-8"); > >> > } catch (Exception exception) { > >> > LOG.error("wrong url code :::" + > >> > exception.getMessage()); > >> > return null; > >> > } > >> > if (url_norm == null) { > >> > LOG.error("cant decode from " + s); > >> > return null; > >> > } > >> > Text to_value = new Text(); > >> > to_value.set(url_norm); > >> > return to_value; > >> > } > >> > } > >> > > >> > > >> > > >> > > >> > >> > >> > >> -- > >> Yours, > >> Zheng > > > > > > > > -- > Yours, > Zheng > >
