[
https://issues.apache.org/jira/browse/HIVE-21052?focusedWorklogId=502802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-502802
]
ASF GitHub Bot logged work on HIVE-21052:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Oct/20 17:25
Start Date: 20/Oct/20 17:25
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #1548:
URL: https://github.com/apache/hive/pull/1548#discussion_r508641433
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -589,4 +593,9 @@ private void checkInterrupt() throws InterruptedException {
throw new InterruptedException("Compaction execution is interrupted");
}
}
-}
+
+ private static boolean isDynPartAbort(Table t, CompactionInfo ci) {
Review comment:
This can be consolidated with most of isDynPartIngest in CompactionUtils
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -400,11 +389,11 @@ public void markCleaned(CompactionInfo info) throws
MetaException {
pStmt.setString(paramCount++, info.partName);
}
if(info.highestWriteId != 0) {
- pStmt.setLong(paramCount++, info.highestWriteId);
+ pStmt.setLong(paramCount, info.highestWriteId);
Review comment:
Why was this changed?
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
##########
@@ -2128,24 +2129,601 @@ public void testCleanerForTxnToWriteId() throws
Exception {
0, TxnDbUtil.countQueryAgent(hiveConf, "select count(*) from
TXN_TO_WRITE_ID"));
}
- private void verifyDirAndResult(int expectedDeltas) throws Exception {
- FileSystem fs = FileSystem.get(hiveConf);
- // Verify the content of subdirs
- FileStatus[] status = fs.listStatus(new Path(TEST_WAREHOUSE_DIR + "/" +
- (Table.MMTBL).toString().toLowerCase()),
FileUtils.HIDDEN_FILES_PATH_FILTER);
+ @Test
+ public void testMmTableAbortWithCompaction() throws Exception {
Review comment:
FYI MM tests are usually in TestTxnCommandsForMmTable.java but I don't
really care about this
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -134,9 +132,6 @@ public CompactionTxnHandler() {
response.add(info);
}
}
-
- LOG.debug("Going to rollback");
- dbConn.rollback();
Review comment:
Any ideas about why this was here? Just curious
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -414,76 +403,30 @@ public void markCleaned(CompactionInfo info) throws
MetaException {
* aborted TXN_COMPONENTS above tc_writeid (and consequently about
aborted txns).
* See {@link ql.txn.compactor.Cleaner.removeFiles()}
*/
- s = "SELECT DISTINCT \"TXN_ID\" FROM \"TXNS\", \"TXN_COMPONENTS\"
WHERE \"TXN_ID\" = \"TC_TXNID\" "
- + "AND \"TXN_STATE\" = " + TxnStatus.ABORTED + " AND
\"TC_DATABASE\" = ? AND \"TC_TABLE\" = ?";
- if (info.highestWriteId != 0) s += " AND \"TC_WRITEID\" <= ?";
- if (info.partName != null) s += " AND \"TC_PARTITION\" = ?";
-
+ s = "DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" IN (" +
Review comment:
This is just refactoring right? LGTM but can you make sure @pvary sees
this as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 502802)
Time Spent: 11h (was: 10h 50m)
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -------------------------------------------------------------------------------------
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 3.0.0, 3.1.1
> Reporter: Jaume M
> Assignee: Jaume M
> Priority: Critical
> Labels: pull-request-available
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch,
> HIVE-21052.8.patch, HIVE-21052.9.patch
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
