[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14715: --- Fix Version/s: 2,1 > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14715: --- Fix Version/s: (was: 2,1) 2.1.1 > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14715: --- Target Version/s: 2.1.1 > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.2.0 > > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14715: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~aihuaxu] for reviewing the code. Push into master branch. > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.2.0 > > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14715: Attachment: HIVE-14715.2.patch > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14715: Status: Patch Available (was: Open) > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14715.1.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14715: Attachment: HIVE-14715.1.patch The NumberFormat Exception throws because of mismatching columns. In genGroupByPlanReduceSinkOperator method, getReduceKeysForReduceSink is called to get Reduce keys from grpByExprs by removing same columns. So reduceKeys.size() <= grpByExprs.size() Reduce values are internal columns which follows grpByExprs, so to get their values, the pos should start from grpByExprs.size(). Attach patch 1 with the fix. > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14715.1.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14715: Description: The java.lang.NumberFormatException will throw with following reproduce: set hive.cbo.enable=false; CREATE TABLE `paqtest`( `c1` int, `s1` string, `s2` string, `bn1` bigint) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; insert into paqtest values (58, '', 'ABC', 0); SELECT 'PM' AS cy, c1, NULL AS iused, NULL AS itp, s2, NULL AS cvg, NULL AS acavg, sum(bn1) AS cca FROM paqtest WHERE (s1 IS NULL OR length(s1) = 0) GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; The stack like following: java.lang.NumberFormatException: ABC GroupByOperator.process(Object, int) line: 773 ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, RawKeyValueIterator, RawComparator, Class, Class) line: 444 ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 Executors$RunnableAdapter.call() line: 471 It works fine when hive.cbo.enable = true was: The java.lang.NumberFormatException will throw with following reproduce: set hive.cbo.enable=false; CREATE TABLE `paqtest`( `c1` int, `s1` string, `s2` string, `bn1` bigint) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; insert into paqtest values (58, '', 'ABC', 0); SELECT 'Pricing mismatch' AS category, c1, NULL AS itemtype_used, NULL AS acq_itemtype, s2, NULL AS currency_used_avg, NULL AS acq_items_avg, sum(bn1) AS cca FROM paqtest WHERE (s1 IS NULL OR length(s1) = 0) GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; The stack like following: java.lang.NumberFormatException: ABC GroupByOperator.process(Object, int) line: 773 ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, RawKeyValueIterator, RawComparator, Class, Class) line: 444 ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 Executors$RunnableAdapter.call() line: 471 It works fine when hive.cbo.enable = true > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
