[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105025#comment-13105025 ] John Sichi commented on HIVE-2182: -- Oops, looks like I typed in the wrong JIRA issue number in the commit message (I typed in HIVE-2181 instead of HIVE-2182), so the Hudson commit message went there instead. I've fixed it in the svn log though. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: 0.9.0 Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.4.patch, HIVE-2182.5.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103803#comment-13103803 ] jirapos...@reviews.apache.org commented on HIVE-2182: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1759/ --- (Updated 2011-09-13 17:32:55.124065) Review request for hive and John Sichi. Changes --- Patch rebased on trunk Summary --- while executing UDF if the implementation jar is not present in MR class path it is throwing nullpointer exception instead of throwing nullpointer exception throwing meaning full exception with the required details. This addresses bug HIVE-2182. https://issues.apache.org/jira/browse/HIVE-2182 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 1170044 trunk/ql/src/test/queries/clientnegative/udfnull.q PRE-CREATION trunk/ql/src/test/results/clientnegative/udfnull.q.out PRE-CREATION trunk/ql/src/test/results/compiler/plan/cast1.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/groupby4.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/groupby6.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/input20.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/input8.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/join2.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1170044 trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1170044 Diff: https://reviews.apache.org/r/1759/diff Testing --- Added unit tests Thanks, chinna Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.4.patch, HIVE-2182.5.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102808#comment-13102808 ] Chinna Rao Lalam commented on HIVE-2182: Sorry...TestParse.java is failing because of the plan xmls. In HIVE-2182.4 updated with the latest plan xmls. Pls check this patch. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.4.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102927#comment-13102927 ] John Sichi commented on HIVE-2182: -- Yeah, I hit those failures too while testing. I'll rerun with the latest patch. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.4.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101396#comment-13101396 ] Chinna Rao Lalam commented on HIVE-2182: Sorry...Now updated the q.out file against the latest trunk Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101405#comment-13101405 ] jirapos...@reviews.apache.org commented on HIVE-2182: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1759/ --- (Updated 2011-09-09 18:08:07.528830) Review request for hive and John Sichi. Summary --- while executing UDF if the implementation jar is not present in MR class path it is throwing nullpointer exception instead of throwing nullpointer exception throwing meaning full exception with the required details. This addresses bug HIVE-2182. https://issues.apache.org/jira/browse/HIVE-2182 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 1167000 trunk/ql/src/test/queries/clientnegative/udfnull.q PRE-CREATION trunk/ql/src/test/results/clientnegative/udfnull.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1759/diff Testing --- Added unit tests Thanks, chinna Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101499#comment-13101499 ] John Sichi commented on HIVE-2182: -- It's still failing for me with the latest patch. Did you use -Doverwrite=true to regenerate the log? {noformat} [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /data/users/jsichi/open/test-trunk/build/ql/test/logs/clientnegative/udfnull.q.out /data/users/jsichi/open/test-trunk/ql/src/test/results/clientnegative/udfnull.q.out [junit] 18c18,27 [junit] /data/users/jsichi/open/test-trunk/build/ql/tmp//hive.log [junit] --- [junit] /home/opensrc/9thsep/build/ql/tmp//hive.log [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] PREHOOK: query: CREATE TEMPORARY FUNCTION example_arraysum AS 'org.apache.hadoop.hive.contrib.udf.example.UDFExampleArraySum' [junit] PREHOOK: type: CREATEFUNCTION [junit] POSTHOOK: query: CREATE TEMPORARY FUNCTION example_arraysum AS 'org.apache.hadoop.hive.contrib.udf.example.UDFExampleArraySum' [junit] POSTHOOK: type: CREATEFUNCTION [junit] PREHOOK: query: SELECT example_arraysum(lint)FROM src_thrift [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@src_thrift [junit] PREHOOK: Output: file:/tmp/root/hive_2011-05-25_10-05-57_126_4632621650656424226/-mr-1 {noformat} Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101501#comment-13101501 ] John Sichi commented on HIVE-2182: -- Oops, sorry, ignore comment above...I misapplied the latest patch. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101508#comment-13101508 ] John Sichi commented on HIVE-2182: -- +1. Will commit when tests pass. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100117#comment-13100117 ] Chinna Rao Lalam commented on HIVE-2182: While de serializing the udfClass variable is becoming null because the class is not present in MR classpath. In user logs it is throwing the following exception java.lang.ClassNotFoundException: com.samples.hive.udf.Grade Continuing ... So i have introduced one new variable to hold the udf class name like udfClassName. Now in the exception the class name will be displayed in the following way Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: The UDF Implementation class 'com.samples.hive.udf.Grade' is Not in class path at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:141) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:896) Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100631#comment-13100631 ] jirapos...@reviews.apache.org commented on HIVE-2182: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1759/ --- Review request for hive and John Sichi. Summary --- while executing UDF if the implementation jar is not present in MR class path it is throwing nullpointer exception instead of throwing nullpointer exception throwing meaning full exception with the required details. This addresses bug HIVE-2182. https://issues.apache.org/jira/browse/HIVE-2182 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 1166865 trunk/ql/src/test/queries/clientnegative/udfnull.q PRE-CREATION trunk/ql/src/test/results/clientnegative/udfnull.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1759/diff Testing --- Added unit tests Thanks, chinna Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094195#comment-13094195 ] John Sichi commented on HIVE-2182: -- I agree with Chinna that in general we don't know whether the client-side class availability matches the MR availability, so this extra check is useful. However, can we improve the diagnostics? At a minimum, we want to know what class is missing; if we can also say which .jar we saw it in on the client-side, that would be great. Code format nitpick: put opening curly brace on same line as if keyword. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092781#comment-13092781 ] Chinna Rao Lalam commented on HIVE-2182: client side validation we can do while creating the temporary function this scenario is already take care in the existing functionality. While creating temporary function it is throwing exception if the UDF jar not in class path(Hive Class path). If the requested jar in client class path and not in the executed MR class path this issue will come example if the UDF jar maintained in the HIVE/LIB and not executed add jar or not updated hive.aux.jars.path this issue will come. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043356#comment-13043356 ] Chinna Rao Lalam commented on HIVE-2182: Hi He Yongqiang, I feel this validation is not possible on client side because the client side validation cannot ensure the jar presence in task tracker classpath. Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[ https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043041#comment-13043041 ] He Yongqiang commented on HIVE-2182: Instead of throwing the exception at runtime after starting the job, can we also do some pre-procssing in the client side to catch this exception? Is that easy to do? Avoid null pointer exception when executing UDF --- Key: HIVE-2182 URL: https://issues.apache.org/jira/browse/HIVE-2182 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2182.patch For using UDF's executed following steps {noformat} add jar /home/udf/udf.jar; create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} But from the above steps if we miss the first step (add jar) and execute remaining steps {noformat} create temporary function grade as 'udf.Grade'; select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m; {noformat} In tasktracker it is throwing this exception {noformat} Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 18 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107) ... 31 more {noformat} Instead of null pointer exception it should throw meaning full exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira