[
https://issues.apache.org/jira/browse/HIVE-26509?focusedWorklogId=812627&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812627
]
ASF GitHub Bot logged work on HIVE-26509:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Sep/22 18:38
Start Date: 27/Sep/22 18:38
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3567:
URL: https://github.com/apache/hive/pull/3567#discussion_r981591719
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -891,33 +855,34 @@ public void run() {
startCondition.await();
}
- if (isLeader) {
- startCompactorInitiator(conf);
- startCompactorCleaner(conf);
- startRemoteOnlyTasks(conf);
- startStatsUpdater(conf);
- HMSHandler.startAlwaysTaskThreads(conf);
- }
-
- // The leader HMS may not necessarily have sufficient compute
capacity required to run
- // actual compaction work. So it can run on a non-leader HMS with
sufficient capacity
- // or a configured HS2 instance.
- if (MetastoreConf.getVar(conf,
MetastoreConf.ConfVars.HIVE_METASTORE_RUNWORKER_IN).equals("metastore")) {
- LOG.warn("Running compaction workers on HMS side is not suggested
because compaction pools are not supported in HMS " +
- "(HIVE-26443). Consider removing the
hive.metastore.runworker.in configuration setting, as it will be " +
- "comletely removed in future releases.");
- startCompactorWorkers(conf);
+ LeaderElectionContext context = new
LeaderElectionContext.ContextBuilder(conf)
+
.setHMSHandler(thriftServer.getHandler()).servHost(getServerHostName())
+ // always tasks
+ .setTType(LeaderElectionContext.TTYPE.ALWAYS_TASKS)
+ .addListener(new HouseKeepingTasks(conf, false))
+ // housekeeping tasks
+ .setTType(LeaderElectionContext.TTYPE.HOUSEKEEPING)
+ .addListener(new CMClearer(conf))
+ .addListener(new StatsUpdaterTask(conf))
+ .addListener(new CompactorTasks(conf, false))
+ .addListener(new CompactorPMF())
+ .addListener(new HouseKeepingTasks(conf, true))
Review Comment:
Ack
Issue Time Tracking
-------------------
Worklog Id: (was: 812627)
Time Spent: 4h 50m (was: 4h 40m)
> Introduce dynamic leader election in HMS
> ----------------------------------------
>
> Key: HIVE-26509
> URL: https://issues.apache.org/jira/browse/HIVE-26509
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
> Reporter: Zhihua Deng
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> From HIVE-21841 we have a leader HMS selected by configuring
> metastore.housekeeping.leader.hostname on startup. This approach saves us
> from running duplicated HMS's housekeeping tasks cluster-wide.
> In this jira, we introduce another dynamic leader election: adopt hive lock
> to implement the leader election. Once a HMS owns the lock, then it becomes
> the leader, carries out the housekeeping tasks, and sends heartbeats to renew
> the lock before timeout. If the leader fails to reclaim the lock, then stops
> the already started tasks if it has, the electing event is audited. We can
> achieve a more dynamic leader when the original goes down or in the public
> cloud without well configured property, and reduce the leader’s burdens by
> running these tasks among different leaders.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
