[
https://issues.apache.org/jira/browse/HIVE-27020?focusedWorklogId=857247&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-857247
]
ASF GitHub Bot logged work on HIVE-27020:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Apr/23 18:12
Start Date: 16/Apr/23 18:12
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #4091:
URL: https://github.com/apache/hive/pull/4091#discussion_r1167991222
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3273,11 +3273,11 @@ public static enum ConfVars {
HIVE_COMPACTOR_ABORTEDTXN_THRESHOLD("hive.compactor.abortedtxn.threshold",
1000,
"Number of aborted transactions involving a given table or partition
that will trigger\n" +
- "a major compaction."),
+ "a major compaction / cleanup of aborted directories."),
Review Comment:
This config is used for controlling the number of aborted txns after which
the cleanup is triggered (either cleanup / compaction) depending on what is
expected.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3273,11 +3273,11 @@ public static enum ConfVars {
HIVE_COMPACTOR_ABORTEDTXN_THRESHOLD("hive.compactor.abortedtxn.threshold",
1000,
"Number of aborted transactions involving a given table or partition
that will trigger\n" +
- "a major compaction."),
+ "a major compaction / cleanup of aborted directories."),
HIVE_COMPACTOR_ABORTEDTXN_TIME_THRESHOLD("hive.compactor.aborted.txn.time.threshold",
"12h",
new TimeValidator(TimeUnit.HOURS),
- "Age of table/partition's oldest aborted transaction when compaction
will be triggered. " +
+ "Age of table/partition's oldest aborted transaction when compaction /
cleanup of aborted directories will be triggered. " +
Review Comment:
This config is used for controlling the time threshold of aborted txns after
which the cleanup is triggered (either cleanup / compaction) depending on what
is expected.
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactorBase.java:
##########
@@ -89,6 +89,7 @@ public void setup() throws Exception {
hiveConf.setBoolVar(HiveConf.ConfVars.HIVEOPTIMIZEMETADATAQUERIES, false);
MetastoreConf.setBoolVar(hiveConf,
MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON, true);
MetastoreConf.setBoolVar(hiveConf,
MetastoreConf.ConfVars.COMPACTOR_CLEANER_ON, true);
+ MetastoreConf.setBoolVar(hiveConf,
MetastoreConf.ConfVars.COMPACTOR_CLEAN_ABORTS_USING_CLEANER, false);
Review Comment:
The plan is to keep this config till some point until we know that abort
cleanup is stable. If there are any issues with the given handler, we can use
the compaction cycle. I will create a task once its determined that this
feature is stable and we can remove this feature flag then.
Issue Time Tracking
-------------------
Worklog Id: (was: 857247)
Time Spent: 12h (was: 11h 50m)
> Implement a separate handler to handle aborted transaction cleanup
> ------------------------------------------------------------------
>
> Key: HIVE-27020
> URL: https://issues.apache.org/jira/browse/HIVE-27020
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sourabh Badhya
> Assignee: Sourabh Badhya
> Priority: Major
> Labels: pull-request-available
> Time Spent: 12h
> Remaining Estimate: 0h
>
> As described in the parent task, once the cleaner is separated into different
> entities, implement a separate handler which can create requests for aborted
> transactions cleanup. This would move the aborted transaction cleanup
> exclusively to the cleaner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
