[
https://issues.apache.org/jira/browse/HIVE-21841?focusedWorklogId=261149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261149
]
ASF GitHub Bot logged work on HIVE-21841:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jun/19 05:16
Start Date: 17/Jun/19 05:16
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #668: HIVE-21841 :
Add configuration to specify the hostname of HMS leader …to run housekeeping
tasks in.
URL: https://github.com/apache/hive/pull/668#discussion_r294129954
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##########
@@ -609,22 +609,10 @@ public void init() throws MetaException {
partitionValidationPattern = null;
}
- // We only initialize once the tasks that need to be run periodically
- if (alwaysThreadsInitialized.compareAndSet(false, true)) {
- ThreadPool.initialize(conf);
- Collection<String> taskNames =
- MetastoreConf.getStringCollection(conf,
ConfVars.TASK_THREADS_ALWAYS);
- for (String taskName : taskNames) {
- MetastoreTaskThread task =
- JavaUtils.newInstance(JavaUtils.getClass(taskName,
MetastoreTaskThread.class));
- task.setConf(conf);
- long freq = task.runFrequency(TimeUnit.MILLISECONDS);
- // For backwards compatibility, since some threads used to be hard
coded but only run if
- // frequency was > 0
- if (freq > 0) {
- ThreadPool.getPool().scheduleAtFixedRate(task, freq, freq,
TimeUnit.MILLISECONDS);
- }
- }
+ // We only initialize once the tasks that need to be run periodically.
For remote metastore
+ // these threads are started along with the other housekeeping threads
only in the leader HMS.
+ if (!isMetaStoreRemote()) {
Review comment:
in the older version, these tasks were created in embedded mode also. Will
it not create backward compatibility issue ? may be another config needs to be
added to control this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 261149)
Time Spent: 50m (was: 40m)
> Leader election in HMS to run housekeeping tasks.
> -------------------------------------------------
>
> Key: HIVE-21841
> URL: https://issues.apache.org/jira/browse/HIVE-21841
> Project: Hive
> Issue Type: New Feature
> Affects Versions: 4.0.0
> Reporter: Ashutosh Bapat
> Assignee: Ashutosh Bapat
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21841.01.patch, HIVE-21841.02.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HMS performs housekeeping tasks. When there are multiple HMSes we need to
> have a leader HMS elected which will carry out those housekeeping tasks.
> These tasks include execution of compaction tasks, auto-discovering
> partitions for external tables, generation of compaction tasks, repl thread
> etc.
> Note that, though the code for compaction tasks, auto-discovery of partitions
> etc. is in Hive, the actual tasks are initiated by an HMS configured to do
> so. So, leader election is required only for HMS and not for HS2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
