[ 
https://issues.apache.org/jira/browse/HIVE-23439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh reassigned HIVE-23439:
-------------------------------------

    Assignee: Rajkumar Singh

> Hive sessions over 24 hours encounter Kerberos-related StatsTask errors
> -----------------------------------------------------------------------
>
>                 Key: HIVE-23439
>                 URL: https://issues.apache.org/jira/browse/HIVE-23439
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, Standalone Metastore
>    Affects Versions: 3.1.0
>            Reporter: Chiran Ravani
>            Assignee: Rajkumar Singh
>            Priority: Critical
>
> We have an application that uses Hive via JDBC. The interesting thing about 
> them is that they have sessions that are established with HiveServer2 for 
> multiple days. After 24 hours, their queries are failing with 
> StatsTask-related errors. From looking in the logs, it looks like the 
> communication breaks down between HiveServer2 and the MetaStore.
> Below is error seen:
> {code}
> 2020-04-22T21:25:53,248 ERROR [Thread-1202599]: exec.StatsTask (:()) - Failed 
> to run stats task
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table 
> tennis. Unable to instantiate 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>  at 
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4927)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:189)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:86)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:82) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch 
> table tennis. Unable to instantiate 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1387) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  ... 6 more
> Caused by: java.lang.RuntimeException: Unable to instantiate 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>  at 
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4790) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4858) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4838) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1378) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298) 
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at 
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
>  ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  ... 6 more
> {code}
> The problem appears to be because of delegation token issued by Hive 
> Metastore could not be renewed by HiveServer2 within 24 hours period of time.
> There is similar issue reported in upstream HIVE-22033 which adderesses 
> similar kind of issue, I backported that fix on my local cluster and deployed 
> the same, but that does not seems to adderess the issue. Problem seems to be 
> GC interval which removed th DT from HMS after expiry which by default is set 
> to 1 hour.
> "hive.cluster.delegation.token.gc-interval", 1, TimeUnit.HOURS, "")



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to