[
https://issues.apache.org/jira/browse/HIVE-23439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-23439:
----------------------------------
Attachment: HIVE-23439.patch
Status: Patch Available (was: Open)
> Hive sessions over 24 hours encounter Kerberos-related StatsTask errors
> -----------------------------------------------------------------------
>
> Key: HIVE-23439
> URL: https://issues.apache.org/jira/browse/HIVE-23439
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
> Affects Versions: 3.1.0
> Reporter: Chiran Ravani
> Assignee: Rajkumar Singh
> Priority: Critical
> Attachments: HIVE-23439.patch
>
>
> We have an application that uses Hive via JDBC. The interesting thing about
> them is that they have sessions that are established with HiveServer2 for
> multiple days. After 24 hours, their queries are failing with
> StatsTask-related errors. From looking in the logs, it looks like the
> communication breaks down between HiveServer2 and the MetaStore.
> Below is error seen:
> {code}
> 2020-04-22T21:25:53,248 ERROR [Thread-1202599]: exec.StatsTask (:()) - Failed
> to run stats task
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
> tennis. Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4927)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:189)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:86)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:82)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
> table tennis. Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1387)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> ... 6 more
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4790)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4858)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4838)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1378)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> ... 6 more
> {code}
> The problem appears to be because of delegation token issued by Hive
> Metastore could not be renewed by HiveServer2 within 24 hours period of time.
> There is similar issue reported in upstream HIVE-22033 which adderesses
> similar kind of issue, I backported that fix on my local cluster and deployed
> the same, but that does not seems to adderess the issue. Problem seems to be
> GC interval which removed th DT from HMS after expiry which by default is set
> to 1 hour.
> "hive.cluster.delegation.token.gc-interval", 1, TimeUnit.HOURS, "")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
