[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in

2015-10-21 Thread Vikram Dixit K (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-11216:
--
Fix Version/s: 1.0.2

> UDF GenericUDFMapKeys throws NPE when a null map value is passed in
> ---
>
> Key: HIVE-11216
> URL: https://issues.apache.org/jira/browse/HIVE-11216
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.2.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Fix For: 1.3.0, 2.0.0, 1.0.2
>
> Attachments: HIVE-11216.1.patch, HIVE-11216.patch
>
>
> We can reproduce the problem as below:
> {noformat}
> hive> show create table map_txt;
> OK
> CREATE  TABLE `map_txt`(
>   `id` int,
>   `content` map)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> ...
> Time taken: 0.233 seconds, Fetched: 18 row(s)
> hive> select * from map_txt;
> OK
> 1   NULL
> Time taken: 0.679 seconds, Fetched: 1 row(s)
> hive> select id, map_keys(content) from map_txt;
> 
> Error during job, obtaining debugging information...
> Examining task ID: task_1435534231122_0025_m_00 (and more) from job 
> job_1435534231122_0025
> Task with the most failures(4):
> -
> Task ID:
>   task_1435534231122_0025_m_00
> URL:
>   
> http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025=task_1435534231122_0025_m_00
> -
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":1,"content":null}
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"id":1,"content":null}
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating 
> map_keys(content)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
> ... 9 more
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
> ... 13 more
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> hive>
> {noformat}
> The error is as below (in mappers):
> {noformat}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at 
> org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113)
> at 
>

[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in

2015-10-21 Thread Vikram Dixit K (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-11216:
--
Fix Version/s: 1.2.2

> UDF GenericUDFMapKeys throws NPE when a null map value is passed in
> ---
>
> Key: HIVE-11216
> URL: https://issues.apache.org/jira/browse/HIVE-11216
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.2.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Fix For: 1.3.0, 2.0.0, 1.0.2, 1.2.2
>
> Attachments: HIVE-11216.1.patch, HIVE-11216.patch
>
>
> We can reproduce the problem as below:
> {noformat}
> hive> show create table map_txt;
> OK
> CREATE  TABLE `map_txt`(
>   `id` int,
>   `content` map)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> ...
> Time taken: 0.233 seconds, Fetched: 18 row(s)
> hive> select * from map_txt;
> OK
> 1   NULL
> Time taken: 0.679 seconds, Fetched: 1 row(s)
> hive> select id, map_keys(content) from map_txt;
> 
> Error during job, obtaining debugging information...
> Examining task ID: task_1435534231122_0025_m_00 (and more) from job 
> job_1435534231122_0025
> Task with the most failures(4):
> -
> Task ID:
>   task_1435534231122_0025_m_00
> URL:
>   
> http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025=task_1435534231122_0025_m_00
> -
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":1,"content":null}
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"id":1,"content":null}
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating 
> map_keys(content)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
> ... 9 more
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
> ... 13 more
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> hive>
> {noformat}
> The error is as below (in mappers):
> {noformat}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at 
> org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113)
> at 
>

[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in

2015-07-09 Thread Yibing Shi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated HIVE-11216:
--
Attachment: HIVE-11216.patch

 UDF GenericUDFMapKeys throws NPE when a null map value is passed in
 ---

 Key: HIVE-11216
 URL: https://issues.apache.org/jira/browse/HIVE-11216
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 1.2.0
Reporter: Yibing Shi
Assignee: Yibing Shi
 Attachments: HIVE-11216.patch


 We can reproduce the problem as below:
 {noformat}
 hive show create table map_txt;
 OK
 CREATE  TABLE `map_txt`(
   `id` int,
   `content` mapint,string)
 ROW FORMAT SERDE
   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
 STORED AS INPUTFORMAT
   'org.apache.hadoop.mapred.TextInputFormat'
 OUTPUTFORMAT
   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
 ...
 Time taken: 0.233 seconds, Fetched: 18 row(s)
 hive select * from map_txt;
 OK
 1   NULL
 Time taken: 0.679 seconds, Fetched: 1 row(s)
 hive select id, map_keys(content) from map_txt;
 
 Error during job, obtaining debugging information...
 Examining task ID: task_1435534231122_0025_m_00 (and more) from job 
 job_1435534231122_0025
 Task with the most failures(4):
 -
 Task ID:
   task_1435534231122_0025_m_00
 URL:
   
 http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025tipid=task_1435534231122_0025_m_00
 -
 Diagnostic Messages for this Task:
 Error: java.lang.RuntimeException: 
 org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
 processing row {id:1,content:null}
 at 
 org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
 Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
 Error while processing row {id:1,content:null}
 at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
 at 
 org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
 ... 8 more
 Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating 
 map_keys(content)
 at 
 org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
 at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
 at 
 org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
 at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
 at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
 ... 9 more
 Caused by: java.lang.NullPointerException
 at 
 org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
 at 
 org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
 ... 13 more
 FAILED: Execution Error, return code 2 from 
 org.apache.hadoop.hive.ql.exec.mr.MapRedTask
 MapReduce Jobs Launched:
 Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
 hive
 {noformat}
 The error is as below (in mappers):
 {noformat}
 Caused by: java.lang.NullPointerException
 at 
 org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
 at 
 org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113)
 at 
 org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:778)
 ... 17 more
 {noformat}
 Looking at the source code:
 {code}
   public Object

[jira] [Updated] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in

2015-07-09 Thread Yibing Shi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated HIVE-11216:
--
Attachment: HIVE-11216.1.patch

Attach a new patch.

 UDF GenericUDFMapKeys throws NPE when a null map value is passed in
 ---

 Key: HIVE-11216
 URL: https://issues.apache.org/jira/browse/HIVE-11216
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 1.2.0
Reporter: Yibing Shi
Assignee: Yibing Shi
 Attachments: HIVE-11216.1.patch, HIVE-11216.patch


 We can reproduce the problem as below:
 {noformat}
 hive show create table map_txt;
 OK
 CREATE  TABLE `map_txt`(
   `id` int,
   `content` mapint,string)
 ROW FORMAT SERDE
   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
 STORED AS INPUTFORMAT
   'org.apache.hadoop.mapred.TextInputFormat'
 OUTPUTFORMAT
   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
 ...
 Time taken: 0.233 seconds, Fetched: 18 row(s)
 hive select * from map_txt;
 OK
 1   NULL
 Time taken: 0.679 seconds, Fetched: 1 row(s)
 hive select id, map_keys(content) from map_txt;
 
 Error during job, obtaining debugging information...
 Examining task ID: task_1435534231122_0025_m_00 (and more) from job 
 job_1435534231122_0025
 Task with the most failures(4):
 -
 Task ID:
   task_1435534231122_0025_m_00
 URL:
   
 http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025tipid=task_1435534231122_0025_m_00
 -
 Diagnostic Messages for this Task:
 Error: java.lang.RuntimeException: 
 org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
 processing row {id:1,content:null}
 at 
 org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
 Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
 Error while processing row {id:1,content:null}
 at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
 at 
 org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
 ... 8 more
 Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating 
 map_keys(content)
 at 
 org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
 at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
 at 
 org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
 at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
 at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
 ... 9 more
 Caused by: java.lang.NullPointerException
 at 
 org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
 at 
 org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
 ... 13 more
 FAILED: Execution Error, return code 2 from 
 org.apache.hadoop.hive.ql.exec.mr.MapRedTask
 MapReduce Jobs Launched:
 Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
 hive
 {noformat}
 The error is as below (in mappers):
 {noformat}
 Caused by: java.lang.NullPointerException
 at 
 org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
 at 
 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
 at 
 org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113)
 at 
 org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:778)
 ... 17 more
 {noformat}
 Looking at the source code: