[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404870#comment-16404870
]
Hive QA commented on HIVE-18975:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915109/HIVE-18975.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 13027 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404774#comment-16404774
]
Hive QA commented on HIVE-18975:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hbase-handler: The patch generated 0 new + 1
unchanged - 80 fixed = 1 total (was 81) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9707/dev-support/hive-personality.sh
|
| git revision | master / 94152c9 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9707/yetus/patch-asflicense-problems.txt
|
| modules | C: hbase-handler U: hbase-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9707/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch, HIVE-18975.2.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
>
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404617#comment-16404617
]
Oleksiy Sayankin commented on HIVE-18975:
-
{quote}
You need fix code style
{quote}
Done.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch, HIVE-18975.2.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
> ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403724#comment-16403724
]
Yongzhi Chen commented on HIVE-18975:
-
Yes, it is. You can use it.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
> ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
>
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403464#comment-16403464
]
Oleksiy Sayankin commented on HIVE-18975:
-
[~ychena] is this correct
[formatter|https://github.com/apache/hive/blob/master/dev-support/eclipse-styles.xml]
for Hive? Can I use it? I have found it
[here|https://cwiki.apache.org/confluence/display/Hive/HowToContribute].
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
> ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403402#comment-16403402
]
Yongzhi Chen commented on HIVE-18975:
-
You need fix code style:
./hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java:251:
if (list.get(i) != null) {: warning: 'if' have incorrect
indentation level 12, expected level should be 10.
./hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java:252:
serialize(list.get(i), eoi, level + 1, ss);: warning: 'if' child
have incorrect indentation level 14, expected level should be 12.
./hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java:253:
}: warning: 'if rcurly' have incorrect indentation level 12,
expected level should be 10.
./hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java:297:
if (list.get(i) != null) {: warning: 'if' have incorrect
indentation level 12, expected level should be 10.
./hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java:298:
serialize(list.get(i), fields.get(i).getFieldObjectInspector(),
level + 1, ss);: warning: 'if' child have incorrect indentation level 14,
expected level should be 12.
./hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java:299:
}: warning: 'if rcurly' have incorrect indentation level 12,
expected level should be 10.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
>
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402648#comment-16402648
]
Hive QA commented on HIVE-18975:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914854/HIVE-18975.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 129 failed/errored test(s), 13819 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_1]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_with_masking]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[source] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_10_trims]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_from_unixtime]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_instr] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union25] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_17]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_script]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_2]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[view_cbo] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_streaming]
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=251)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=153)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_convert]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_truncate_cols]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation0]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_create_table_does_not_exist]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad_db1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tablestatus]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_out_of_range]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=95)
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402551#comment-16402551
]
Hive QA commented on HIVE-18975:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-handler: The patch generated 6 new + 79
unchanged - 2 fixed = 85 total (was 81) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9670/dev-support/hive-personality.sh
|
| git revision | master / 6218d8a |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9670/yetus/diff-checkstyle-hbase-handler.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9670/yetus/patch-asflicense-problems.txt
|
| modules | C: hbase-handler U: hbase-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9670/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
>
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402517#comment-16402517
]
Yongzhi Chen commented on HIVE-18975:
-
The change looks fine. +1
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
> ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
>
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402301#comment-16402301
]
Ashutosh Chauhan commented on HIVE-18975:
-
+1 pending test.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
> ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
>
[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table
[
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401790#comment-16401790
]
Oleksiy Sayankin commented on HIVE-18975:
-
[~ychena], [~Yibing], [~aihuaxu] please review.
> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
> `id` INT,
> `map_column` STRUCT) ROW FORMAT
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
> ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
>
