[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
[ https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-11216: -- Fix Version/s: 1.2.2 > UDF GenericUDFMapKeys throws NPE when a null map value is passed in > --- > > Key: HIVE-11216 > URL: https://issues.apache.org/jira/browse/HIVE-11216 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.2.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Fix For: 1.3.0, 2.0.0, 1.0.2, 1.2.2 > > Attachments: HIVE-11216.1.patch, HIVE-11216.patch > > > We can reproduce the problem as below: > {noformat} > hive> show create table map_txt; > OK > CREATE TABLE `map_txt`( > `id` int, > `content` map) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' > ... > Time taken: 0.233 seconds, Fetched: 18 row(s) > hive> select * from map_txt; > OK > 1 NULL > Time taken: 0.679 seconds, Fetched: 1 row(s) > hive> select id, map_keys(content) from map_txt; > > Error during job, obtaining debugging information... > Examining task ID: task_1435534231122_0025_m_00 (and more) from job > job_1435534231122_0025 > Task with the most failures(4): > - > Task ID: > task_1435534231122_0025_m_00 > URL: > > http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025&tipid=task_1435534231122_0025_m_00 > - > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180) > ... 8 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating > map_keys(content) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) > ... 13 more > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > hive> > {noformat} > The error is as below (in mappers): > {noformat} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113) > at > org.apac
[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
[ https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-11216: -- Fix Version/s: 1.0.2 > UDF GenericUDFMapKeys throws NPE when a null map value is passed in > --- > > Key: HIVE-11216 > URL: https://issues.apache.org/jira/browse/HIVE-11216 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.2.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Fix For: 1.3.0, 2.0.0, 1.0.2 > > Attachments: HIVE-11216.1.patch, HIVE-11216.patch > > > We can reproduce the problem as below: > {noformat} > hive> show create table map_txt; > OK > CREATE TABLE `map_txt`( > `id` int, > `content` map) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' > ... > Time taken: 0.233 seconds, Fetched: 18 row(s) > hive> select * from map_txt; > OK > 1 NULL > Time taken: 0.679 seconds, Fetched: 1 row(s) > hive> select id, map_keys(content) from map_txt; > > Error during job, obtaining debugging information... > Examining task ID: task_1435534231122_0025_m_00 (and more) from job > job_1435534231122_0025 > Task with the most failures(4): > - > Task ID: > task_1435534231122_0025_m_00 > URL: > > http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025&tipid=task_1435534231122_0025_m_00 > - > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180) > ... 8 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating > map_keys(content) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) > ... 13 more > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > hive> > {noformat} > The error is as below (in mappers): > {noformat} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113) > at > org.apache.hado
[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
[ https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated HIVE-11216: -- Attachment: HIVE-11216.1.patch Attach a new patch. > UDF GenericUDFMapKeys throws NPE when a null map value is passed in > --- > > Key: HIVE-11216 > URL: https://issues.apache.org/jira/browse/HIVE-11216 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.2.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: HIVE-11216.1.patch, HIVE-11216.patch > > > We can reproduce the problem as below: > {noformat} > hive> show create table map_txt; > OK > CREATE TABLE `map_txt`( > `id` int, > `content` map) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' > ... > Time taken: 0.233 seconds, Fetched: 18 row(s) > hive> select * from map_txt; > OK > 1 NULL > Time taken: 0.679 seconds, Fetched: 1 row(s) > hive> select id, map_keys(content) from map_txt; > > Error during job, obtaining debugging information... > Examining task ID: task_1435534231122_0025_m_00 (and more) from job > job_1435534231122_0025 > Task with the most failures(4): > - > Task ID: > task_1435534231122_0025_m_00 > URL: > > http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025&tipid=task_1435534231122_0025_m_00 > - > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180) > ... 8 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating > map_keys(content) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) > ... 13 more > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > hive> > {noformat} > The error is as below (in mappers): > {noformat} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113) > at > org.apache.hadoop.hive.ql.exec.GroupB
[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
[ https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated HIVE-11216: -- Attachment: HIVE-11216.patch > UDF GenericUDFMapKeys throws NPE when a null map value is passed in > --- > > Key: HIVE-11216 > URL: https://issues.apache.org/jira/browse/HIVE-11216 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.2.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: HIVE-11216.patch > > > We can reproduce the problem as below: > {noformat} > hive> show create table map_txt; > OK > CREATE TABLE `map_txt`( > `id` int, > `content` map) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' > ... > Time taken: 0.233 seconds, Fetched: 18 row(s) > hive> select * from map_txt; > OK > 1 NULL > Time taken: 0.679 seconds, Fetched: 1 row(s) > hive> select id, map_keys(content) from map_txt; > > Error during job, obtaining debugging information... > Examining task ID: task_1435534231122_0025_m_00 (and more) from job > job_1435534231122_0025 > Task with the most failures(4): > - > Task ID: > task_1435534231122_0025_m_00 > URL: > > http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025&tipid=task_1435534231122_0025_m_00 > - > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row {"id":1,"content":null} > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180) > ... 8 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating > map_keys(content) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) > ... 13 more > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > hive> > {noformat} > The error is as below (in mappers): > {noformat} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) > at > org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:77