[
https://issues.apache.org/jira/browse/HIVE-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000995#comment-15000995
]
Eugene Koifman commented on HIVE-11388:
---------------------------------------
A simpler way to deal with Cleaner: entries in COMPACTION_QUEUE have WORKER_ID
which includes hostname of the worker. If cleaner runs on each metastore, we
can make each Cleaner only handle entries from Worker(s) from the same host.
This will ensure Cleaner don't step on each other.
Side note, when metastores get restarted (perhaps even on different host), we
have logic to removeTimedoutWokers() so we can piggyback on this to ensure that
Cleaner doesn't miss any compactions (in the long run at least). But also see
HIVE-11685.
We also want to make sure AcidHouseKeeper instances don't step on each other.
> there should only be 1 Initiator for compactions per Hive installation
> ----------------------------------------------------------------------
>
> Key: HIVE-11388
> URL: https://issues.apache.org/jira/browse/HIVE-11388
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 1.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
>
> org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs
> inside the metastore service to manage compactions of ACID tables. There
> should be exactly 1 instance of this thread (even with multiple Thrift
> services).
> This is documented in
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
> but not enforced.
> Should add enforcement, since more than 1 Initiator could cause concurrent
> attempts to compact the same table/partition - which will not work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
