[
https://issues.apache.org/jira/browse/HIVE-23439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123346#comment-17123346
]
Hive QA commented on HIVE-23439:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004548/HIVE-23439.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17222 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.allocateNextWriteIdRetriesAfterDetectingConflictingConcurrentInsert
(batchId=246)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testExternalTablesReplLoadBootstrapIncr
(batchId=207)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22706/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22706/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22706/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004548 - PreCommit-HIVE-Build
> Hive sessions over 24 hours encounter Kerberos-related StatsTask errors
> -----------------------------------------------------------------------
>
> Key: HIVE-23439
> URL: https://issues.apache.org/jira/browse/HIVE-23439
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
> Affects Versions: 3.1.0
> Reporter: Chiran Ravani
> Assignee: Rajkumar Singh
> Priority: Critical
> Attachments: HIVE-23439.patch
>
>
> We have an application that uses Hive via JDBC. The interesting thing about
> them is that they have sessions that are established with HiveServer2 for
> multiple days. After 24 hours, their queries are failing with
> StatsTask-related errors. From looking in the logs, it looks like the
> communication breaks down between HiveServer2 and the MetaStore.
> Below is error seen:
> {code}
> 2020-04-22T21:25:53,248 ERROR [Thread-1202599]: exec.StatsTask (:()) - Failed
> to run stats task
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
> tennis. Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4927)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:189)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:86)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:82)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
> table tennis. Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1387)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> ... 6 more
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4790)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4858)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4838)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1378)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
> ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> ... 6 more
> {code}
> The problem appears to be because of delegation token issued by Hive
> Metastore could not be renewed by HiveServer2 within 24 hours period of time.
> There is similar issue reported in upstream HIVE-22033 which adderesses
> similar kind of issue, I backported that fix on my local cluster and deployed
> the same, but that does not seems to adderess the issue. Problem seems to be
> GC interval which removed th DT from HMS after expiry which by default is set
> to 1 hour.
> "hive.cluster.delegation.token.gc-interval", 1, TimeUnit.HOURS, "")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
