[
https://issues.apache.org/jira/browse/HIVE-23873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161674#comment-17161674
]
Chiran Ravani edited comment on HIVE-23873 at 7/21/20, 3:09 AM:
----------------------------------------------------------------
[~srahman] Thanks for adding more light to it. I am now able to further narrow
down the issue with CBO. The problem appears only when CBO is off. Below is
the query run with CBO on for Derby.
{code}
20/07/20 19:49:18 [787073ad-c44e-4e04-8cfb-ba49e14602a0 main]: INFO
dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT "IKEY", "bkey",
"fkey", "dkey"
FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1"]
20/07/20 19:49:18 [Hive Hook Proto Log Writer 0]: INFO zlib.ZlibFactory:
Successfully loaded & initialized native-zlib library
20/07/20 19:49:18 [Hive Hook Proto Log Writer 0]: INFO compress.CodecPool: Got
brand-new compressor [.deflate]
20/07/20 19:49:18 [787073ad-c44e-4e04-8cfb-ba49e14602a0 main]: INFO
jdbc.JdbcSerDe: **** Blob data = {dkey=OW[class=class
java.lang.Double,value=20.0], bkey=OW[class=class java.lang.Long,value=20],
fkey=OW[class=class java.lang.Float,value=20.0], IKEY=OW[class=class
java.lang.Integer,value=20]}
{code}
When turning off CBO, query picked is Select * and problem appears.
{code}
20/07/21 03:08:44 [ce531fd3-ca3f-4dc4-8681-3cf1233f27df main]: INFO
jdbc.JdbcInputFormat: Creating 1 input split limit:-1, offset:0
20/07/21 03:08:44 [ce531fd3-ca3f-4dc4-8681-3cf1233f27df main]: INFO
dao.GenericJdbcDatabaseAccessor: Query to execute is [select * from
EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1]
Failed with exception java.io.IOException:java.lang.NullPointerException
20/07/21 03:08:44 [ce531fd3-ca3f-4dc4-8681-3cf1233f27df main]: ERROR CliDriver:
Failed with exception java.io.IOException:java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
.
.
Caused by: java.lang.NullPointerException
at
org.apache.hive.storage.jdbc.JdbcSerDe.deserialize(JdbcSerDe.java:235)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:619)
... 17 more
{code}
With Oracle, the CBO fails and skipped causing the problem, below is my trace.
{code}
20/07/21 02:13:18 [Heartbeater-0]: INFO metastore.RetryingMetaStoreClient:
RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
ugi=h...@example.com (auth:KERBEROS) retries=24 delay=5 lifetime=0
20/07/21 02:14:31 [f68e0ab8-1585-4db5-a872-20790c5eb3dd main]: ERROR
parse.CalcitePlanner: CBO failed, skipping CBO.
java.lang.IllegalArgumentException: Multiple entries with same key:
APEX_ACTIVITY_LOG=JdbcTable {APEX_ACTIVITY_LOG} and APEX_ACTIVITY_LOG=JdbcTable
{APEX_ACTIVITY_LOG}
at
org.apache.hive.com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.adapter.jdbc.JdbcSchema.computeTables(JdbcSchema.java:295)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.adapter.jdbc.JdbcSchema.getTableMap(JdbcSchema.java:351)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.adapter.jdbc.JdbcSchema.getTable(JdbcSchema.java:345)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3203)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5182)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1849)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1795)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1556)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:541)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12460)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:435)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:290)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:166)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:402)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:351)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:345)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:129)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:231)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:862)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:798)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:717)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_112]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_112]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
~[hadoop-common-3.1.1.3.1.5.5-2.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
~[hadoop-common-3.1.1.3.1.5.5-2.jar:?]
20/07/21 02:14:31 [f68e0ab8-1585-4db5-a872-20790c5eb3dd main]: INFO
parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
20/07/21 02:14:31 [f68e0ab8-1585-4db5-a872-20790c5eb3dd main]: INFO
parse.CalcitePlanner: Get metadata for source tables
{code}
With that, I think problem with Oracle CBO should be fixed. I will investigate
the issue ob what could be causing such problem.
Also, maybe we can fix for condition when CBO is disabled or for any reason CBO
is skipped. Attaching second patch.
was (Author: chiran54321):
[~srahman] Thanks for adding more light to it. I am now able to further narrow
down the issue with CBO. The problem appears only when CBO is off. Below is
the query run with CBO on for Derby.
{code}
20/07/20 19:49:18 [787073ad-c44e-4e04-8cfb-ba49e14602a0 main]: INFO
dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT "IKEY", "bkey",
"fkey", "dkey"
FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1"]
20/07/20 19:49:18 [Hive Hook Proto Log Writer 0]: INFO zlib.ZlibFactory:
Successfully loaded & initialized native-zlib library
20/07/20 19:49:18 [Hive Hook Proto Log Writer 0]: INFO compress.CodecPool: Got
brand-new compressor [.deflate]
20/07/20 19:49:18 [787073ad-c44e-4e04-8cfb-ba49e14602a0 main]: INFO
jdbc.JdbcSerDe: **** Blob data = {dkey=OW[class=class
java.lang.Double,value=20.0], bkey=OW[class=class java.lang.Long,value=20],
fkey=OW[class=class java.lang.Float,value=20.0], IKEY=OW[class=class
java.lang.Integer,value=20]}
{code}
When turning off CBO, query picked is Select * and problem appears.
{code}
20/07/20 20:00:56 [bfdb833e-8366-451c-86ef-f38e2b002690 main]: INFO
dao.GenericJdbcDatabaseAccessor: Query to execute is [select * from
TESTHIVEJDBCSTORAGE]
20/07/20 20:00:56 [bfdb833e-8366-451c-86ef-f38e2b002690 main]: INFO
jdbc.JdbcSerDe: **** Blob data = {fname=OW[class=class
java.lang.String,value=Name1], id=OW[class=class java.lang.Integer,value=1]}
{code}
With Oracle, the CBO fails and skipped causing the problem, below is my trace.
{code}
20/07/21 02:13:18 [Heartbeater-0]: INFO metastore.RetryingMetaStoreClient:
RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
ugi=h...@example.com (auth:KERBEROS) retries=24 delay=5 lifetime=0
20/07/21 02:14:31 [f68e0ab8-1585-4db5-a872-20790c5eb3dd main]: ERROR
parse.CalcitePlanner: CBO failed, skipping CBO.
java.lang.IllegalArgumentException: Multiple entries with same key:
APEX_ACTIVITY_LOG=JdbcTable {APEX_ACTIVITY_LOG} and APEX_ACTIVITY_LOG=JdbcTable
{APEX_ACTIVITY_LOG}
at
org.apache.hive.com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.adapter.jdbc.JdbcSchema.computeTables(JdbcSchema.java:295)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.adapter.jdbc.JdbcSchema.getTableMap(JdbcSchema.java:351)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.adapter.jdbc.JdbcSchema.getTable(JdbcSchema.java:345)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3203)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5182)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1849)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1795)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1556)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:541)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12460)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:435)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:290)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:166)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:402)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:351)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:345)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:129)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:231)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:862)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:798)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:717)
~[hive-cli-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_112]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_112]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
~[hadoop-common-3.1.1.3.1.5.5-2.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
~[hadoop-common-3.1.1.3.1.5.5-2.jar:?]
20/07/21 02:14:31 [f68e0ab8-1585-4db5-a872-20790c5eb3dd main]: INFO
parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
20/07/21 02:14:31 [f68e0ab8-1585-4db5-a872-20790c5eb3dd main]: INFO
parse.CalcitePlanner: Get metadata for source tables
{code}
With that, I think problem with Oracle CBO should be fixed. I will investigate
the issue ob what could be causing such problem.
Also, maybe we can fix for condition when CBO is disabled or for any reason CBO
is skipped. Attaching second patch.
> Querying Hive JDBCStorageHandler table fails with NPE
> -----------------------------------------------------
>
> Key: HIVE-23873
> URL: https://issues.apache.org/jira/browse/HIVE-23873
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
> Affects Versions: 3.1.0, 3.1.1, 3.1.2
> Reporter: Chiran Ravani
> Assignee: Chiran Ravani
> Priority: Critical
> Attachments: HIVE-23873.01.patch, HIVE-23873.02.patch
>
>
> Scenario is Hive table having same schema as table in Oracle, however when we
> query the table with data it fails with NPE, below is the trace.
> {code}
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:617)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:524)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473)
> ~[hive-service-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> ... 34 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.deserialize(JdbcSerDe.java:164)
> ~[hive-jdbc-handler-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:598)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:524)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473)
> ~[hive-service-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
> ... 34 more
> {code}
> Problem appears when column names in Oracle are in Upper case and since in
> Hive, table and column names are forced to store in lowercase during
> creation. User runs into NPE error while fetching data.
> While deserializing data, input consists of column names in lower case which
> fails to get the value
> https://github.com/apache/hive/blob/rel/release-3.1.2/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java#L136
> {code}
> rowVal = ((ObjectWritable)value).get();
> {code}
> Log Snio:
> =============
> {code}
> 2020-07-17T16:49:09,598 INFO [04ed42ec-91d2-4662-aee7-37e840a06036
> HiveServer2-Handler-Pool: Thread-104]: dao.GenericJdbcDatabaseAccessor (:())
> - Query to execute is [select * from TESTHIVEJDBCSTORAGE]
> 2020-07-17T16:49:10,642 INFO [04ed42ec-91d2-4662-aee7-37e840a06036
> HiveServer2-Handler-Pool: Thread-104]: jdbc.JdbcSerDe (:()) - *** ColumnKey =
> ID
> 2020-07-17T16:49:10,642 INFO [04ed42ec-91d2-4662-aee7-37e840a06036
> HiveServer2-Handler-Pool: Thread-104]: jdbc.JdbcSerDe (:()) - *** Blob value
> = {fname=OW[class=class java.lang.String,value=Name1], id=OW[class=class
> java.lang.Integer,value=1]}
> {code}
> Simple Reproducer for this case.
> =============
> 1. Create table in Oracle
> {code}
> create table TESTHIVEJDBCSTORAGE(ID INT, FNAME VARCHAR(20));
> {code}
> 2. Insert dummy data.
> {code}
> Insert into TESTHIVEJDBCSTORAGE values (1, 'Name1');
> {code}
> 3. Create JDBCStorageHandler table in Hive.
> {code}
> CREATE EXTERNAL TABLE default.TESTHIVEJDBCSTORAGE_HIVE_TBL (ID INT, FNAME
> VARCHAR(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "ORACLE",
> "hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
> "hive.sql.jdbc.url" = "jdbc:oracle:thin:@orachehostname/XE",
> "hive.sql.dbcp.username" = "chiran",
> "hive.sql.dbcp.password" = "supersecurepassword",
> "hive.sql.table" = "TESTHIVEJDBCSTORAGE",
> "hive.sql.dbcp.maxActive" = "1"
> );
> {code}
> 4. Query Hive table, fails with NPE.
> {code}
> > select * from default.TESTHIVEJDBCSTORAGE_HIVE_TBL;
> INFO : Compiling
> command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc):
> select * from default.TESTHIVEJDBCSTORAGE_HIVE_TBL
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:testhivejdbcstorage_hive_tbl.id,
> type:int, comment:null), FieldSchema(name:testhivejdbcstorage_hive_tbl.fname,
> type:varchar(20), comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc);
> Time taken: 9.914 seconds
> INFO : Executing
> command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc):
> select * from default.TESTHIVEJDBCSTORAGE_HIVE_TBL
> INFO : Completed executing
> command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc);
> Time taken: 0.019 seconds
> INFO : OK
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {code}
> Assuming that there are no repercussions, can we convert the column names to
> lowercase fetched from Database/Query pointing to table in JDBCStorageHandler?
> Attaching the patch for the case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
