[
https://issues.apache.org/jira/browse/HIVE-29720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097796#comment-18097796
]
arun kumar commented on HIVE-29720:
-----------------------------------
Hi [~dengzh] , I'd like to pick this up if it isn't already being worked on.
One thing I noticed reading HIVE-26509: retry exhaustion in
\{{LeaseLeaderElection.tryBeLeader}} throws \{{LeaderException}}, which
\{{LeaderElectionContext.start()}} rewraps as \{{RuntimeException}} inside a
daemon thread that has no \{{UncaughtExceptionHandler}}. So today the failure
isn't warned — the daemon thread dies silently and HMS keeps running with that
\{{TTYPE}} unelected. Matches the symptoms in the description.
Planned approach: add an \{{UncaughtExceptionHandler}} on the election daemon
that logs + audits the failure, then triggers an orderly HMS shutdown. Existing
\{{metastore.lock.numretries}} / \{{metastore.lock.sleep.between.retries}}
knobs stay as the tolerance dial.
Two questions before I start:
# Orderly shutdown, or hard \{{Runtime.halt()}} from the daemon?
# {\{HOUSEKEEPING}} and \{{ALWAYS_TASKS}} elect independently — should abort
trigger on any failure, or only specific \{{TTYPE}}s?
Thanks!
> Abort the service if Metastore cannot participate in the election
> -----------------------------------------------------------------
>
> Key: HIVE-29720
> URL: https://issues.apache.org/jira/browse/HIVE-29720
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Reporter: Zhihua Deng
> Priority: Major
>
> For the lease leader election, if a Metastore cannot participate in the
> election within retries, it's still running with only a message warned. This
> could be an issue if all Metastores have failed the election or a restarted
> leader cannot join in the election anymore due to some unknown exception. As
> a consequence no housekeeping tasks are running across the warehouse, and the
> user cannot figure it out early until things become wrong, such as normal
> queries become slow due to no compaction over the delta files. It would be
> nice to abort the Metastore early before things become worse.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
