Chiran Ravani created HIVE-23439:
------------------------------------
Summary: Hive sessions over 24 hours encounter Kerberos-related
StatsTask errors
Key: HIVE-23439
URL: https://issues.apache.org/jira/browse/HIVE-23439
Project: Hive
Issue Type: Bug
Components: HiveServer2, Standalone Metastore
Affects Versions: 3.1.0
Reporter: Chiran Ravani
We have an application that uses Hive via JDBC. The interesting thing about
them is that they have sessions that are established with HiveServer2 for
multiple days. After 24 hours, their queries are failing with StatsTask-related
errors. From looking in the logs, it looks like the communication breaks down
between HiveServer2 and the MetaStore.
Below is error seen:
{code}
2020-04-22T21:25:53,248 ERROR [Thread-1202599]: exec.StatsTask (:()) - Failed
to run stats task
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table tennis.
Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4927)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:189)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:86)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:82)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
table tennis. Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1387)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
... 6 more
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4790)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4858)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4838)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1378)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
at
org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
... 6 more
{code}
The problem appears to be because of delegation token issued by Hive Metastore
could not be renewed by HiveServer2 within 24 hours period of time.
There is similar issue reported in upstream HIVE-22033 which adderesses similar
kind of issue, I backported that fix on my local cluster and deployed the same,
but that does not seems to adderess the issue. Problem seems to be GC interval
which removed th DT from HMS after expiry which by default is set to 1 hour.
"hive.cluster.delegation.token.gc-interval", 1, TimeUnit.HOURS, "")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
