[
https://issues.apache.org/jira/browse/HIVE-27117?focusedWorklogId=849213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-849213
]
ASF GitHub Bot logged work on HIVE-27117:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Mar/23 08:18
Start Date: 06/Mar/23 08:18
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #4096:
URL: https://github.com/apache/hive/pull/4096#discussion_r1126069879
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -3647,9 +3647,12 @@ long generateCompactionQueueId(Statement stmt) throws
SQLException, MetaExceptio
+ "no record found in next_compaction_queue_id");
}
long id = rs.getLong(1);
- s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1);
+ s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1)
+ " WHERE \"NCQ_NEXT\" = " + id;
LOG.debug("Going to execute update <{}>", s);
- stmt.executeUpdate(s);
+ if (stmt.executeUpdate(s) != 1) {
Review Comment:
This is a short term quick and dirty fix, as the id generation for
compaction requests is broken. The old approach did not guarantee a unique id
at all, and since Initiator processes the candidates in a parallel way, this
could be a production issue too. Actually I have seen this duplicate key error
in production logs before. The long term solution is to modify the id to auto
increment field. It is already done this way for TXNS table. However, this
change will be bigger, therefore I created a separate task for it:
https://issues.apache.org/jira/browse/HIVE-27121
Issue Time Tracking
-------------------
Worklog Id: (was: 849213)
Time Spent: 1h 20m (was: 1h 10m)
> Fix compaction related flaky tests
> ----------------------------------
>
> Key: HIVE-27117
> URL: https://issues.apache.org/jira/browse/HIVE-27117
> Project: Hive
> Issue Type: Task
> Reporter: László Végh
> Assignee: László Végh
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The following tests turned out to be flaky recently:
> *
> org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDropTableAndCompactionConcurrent
> *
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testInitiatorFailuresCountedCorrectly
> *
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMajorCompactionNotPartitionedWithoutBuckets
> *
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testCompactionWithCreateTableProps
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
