This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 16d379072f2 [fix](txn) Release the routine load job lock when a
transaction abort fails (#70178)
16d379072f2 is described below
commit 16d379072f2447d5e7d2feefe5806d59cb70ec3e
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Nov 20 15:31:29 2026 +0800
[fix](txn) Release the routine load job lock when a transaction abort fails
(#70178)
### Background
In shared-nothing mode, transaction jobs receive `beforeAborted` and
`afterAborted` callbacks around an abort. A routine load job takes its
write lock in the first callback and releases it in the second, so every
successful before callback needs a matching after callback. Delete jobs
also use these callbacks to release their registration, or retain it
until a committed transaction becomes visible. Cloud transaction
callbacks are delivered by the cloud transaction manager after its Meta
Service RPC.
### What problem does this PR solve?
Issue Number: close #69961
Related PR: #59990 (introduced the lock leak), #68514 (restored commit
callback pairing)
PR Type: Bugfix
Problem Summary:
Since #59990, rejecting an abort of a COMMITTED routine load transaction
or failing its edit-log write leaves the job lock held: job commands,
scheduling and transaction publishing can hang until FE restarts. Both
abort entry points now pair the callbacks in a `finally` outside the
transaction monitor while preserving the original exception. A rejected
shared-nothing abort keeps the delete callback needed for publication;
cloud aborts keep their existing cleanup, including when Meta Service
already aborted the transaction. Most of the diff is tests.
### Details
- **Change map:** Production code: +38/-17 lines; tests: +251/-0;
generated files and docs: 0. Callback pairing and exception handling in
the two local abort entry points account for 49 changed lines;
mode-specific delete callback ownership accounts for 6. The 251 test
lines cover lock release, exception handling, publication and
local/cloud delete callback cleanup.
- **Frequency:** Not observed in production; found by code review.
Triggers on a late abort of a COMMITTED routine load transaction or an
edit-log failure.
- **Root cause:** #59990 moved abort processing under per-transaction
monitors and dropped the `finally` that ran `afterStateTransform`.
Rejected state transitions and edit-log failures therefore bypassed the
callback that releases the job lock.
- **Why existing tests missed it:** Unit tests covered aborts without a
routine-load callback and callback pairing only on commit. Cloud abort
tests registered no real DeleteJob, so they could not detect retained
delete callbacks. Regression suites for routine load and `delete_p0`
exercise ordinary transactions and SQL results, without forcing a
COMMITTED abort, a log failure, or checking callback ownership after an
MS timeout abort.
- **Fix:** Run the after callback from a `finally` outside the
transaction monitor and log callback failures separately. Retain an
unoperated delete callback only in shared-nothing mode: cloud
`txnOperated=false` can mean MS has already aborted the transaction, so
its callback must still be removed.
- **Other paths and branches:** Both normal and 2PC local aborts are
covered. Commit pairing was fixed in #68514; cloud already pairs abort
callbacks in `finally`. `branch-3.1`, `branch-4.0`, `branch-4.1` and
`branch-4.2` retain the original finally pairing and unconditional
delete cleanup, so no backport is needed.
- **Behavior change:** A failing after callback is logged instead of
turning a successful abort into a failed request, as on commit and
visible paths. The original abort failure still propagates.
### Testing
- Locally ran `run-fe-ut.sh --run` over `CloudGlobalTransactionMgrTest`,
`DatabaseTransactionMgrTest`, `GlobalTransactionMgrTest`,
`TransactionStateTest`, `DeleteJobTest` and `RoutineLoadJobTest`: 152
tests, 0 failures, 0 errors, 0 skipped. This includes the existing
rejected-abort lock, exception-priority, publication and shared-nothing
delete callback tests.
- New `testAbortDeleteTransactionRemovesCallback` covers both `OK` and
`TXN_ALREADY_ABORTED` through the cloud abort entry point with a real
DeleteJob. On reviewed commit `b878321` plus the test, the
already-aborted case fails because the callback remains registered; both
cases pass with this fix.
- The cloud cases mock MS RPC responses. A real FE/MS recycler timeout
scenario was not run.
---
.../main/java/org/apache/doris/load/DeleteJob.java | 6 +-
.../doris/transaction/DatabaseTransactionMgr.java | 49 +++--
.../transaction/CloudGlobalTransactionMgrTest.java | 48 +++++
.../transaction/DatabaseTransactionMgrTest.java | 203 +++++++++++++++++++++
4 files changed, 289 insertions(+), 17 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
index 51ff705aebd..94c3cee6fda 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
@@ -241,7 +241,11 @@ public class DeleteJob extends
AbstractTxnStateChangeCallback implements DeleteJ
@Override
public void afterAborted(TransactionState txnState, boolean txnOperated,
String txnStatusChangeReason) {
- // just to clean the callback
+ if (!txnOperated && !Config.isCloudMode()) {
+ // A rejected shared-nothing abort may still become visible and
needs afterVisible().
+ return;
+ }
+ // Cloud aborts must also clean up transactions that MS already
aborted (txnOperated=false).
Env.getCurrentGlobalTransactionMgr().getCallbackFactory().removeCallback(getId());
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
index 2d37d7b0bde..0107f4212f3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
@@ -2050,17 +2050,26 @@ public class DatabaseTransactionMgr {
transactionState.beforeStateTransform(TransactionStatus.ABORTED);
boolean txnOperated = false;
EditLog.EditLogItem logItem = null;
- synchronized (transactionState) {
- txnOperated = unprotectAbortTransaction(transactionId, reason);
- if (txnOperated) {
- if (Config.enable_txn_log_outside_lock) {
- logItem = enqueueTransactionState(transactionState);
- } else {
- persistTransactionState(transactionState);
+ try {
+ synchronized (transactionState) {
+ txnOperated = unprotectAbortTransaction(transactionId, reason);
+ if (txnOperated) {
+ if (Config.enable_txn_log_outside_lock) {
+ logItem = enqueueTransactionState(transactionState);
+ } else {
+ persistTransactionState(transactionState);
+ }
}
}
+ } finally {
+ // Always pair a successful before callback outside the
transaction monitor, also when the abort
+ // is rejected or its edit log fails: a routine load job releases
the lock taken in beforeAborted here.
+ try {
+
transactionState.afterStateTransform(TransactionStatus.ABORTED, txnOperated,
reason);
+ } catch (Throwable e) {
+ LOG.warn("afterStateTransform txn {} failed. exception: ",
transactionState, e);
+ }
}
- transactionState.afterStateTransform(TransactionStatus.ABORTED,
txnOperated, reason);
if (txnOperated) {
awaitTransactionState(logItem, transactionState);
}
@@ -2100,17 +2109,25 @@ public class DatabaseTransactionMgr {
transactionState.beforeStateTransform(TransactionStatus.ABORTED);
boolean txnOperated = false;
EditLog.EditLogItem logItem = null;
- synchronized (transactionState) {
- txnOperated = unprotectAbortTransaction(transactionId, "User
Abort");
- if (txnOperated) {
- if (Config.enable_txn_log_outside_lock) {
- logItem = enqueueTransactionState(transactionState);
- } else {
- persistTransactionState(transactionState);
+ try {
+ synchronized (transactionState) {
+ txnOperated = unprotectAbortTransaction(transactionId, "User
Abort");
+ if (txnOperated) {
+ if (Config.enable_txn_log_outside_lock) {
+ logItem = enqueueTransactionState(transactionState);
+ } else {
+ persistTransactionState(transactionState);
+ }
}
}
+ } finally {
+ // Always pair a successful before callback outside the
transaction monitor, as in abortTransaction().
+ try {
+
transactionState.afterStateTransform(TransactionStatus.ABORTED, txnOperated,
"User Abort");
+ } catch (Throwable e) {
+ LOG.warn("afterStateTransform txn {} failed. exception: ",
transactionState, e);
+ }
}
- transactionState.afterStateTransform(TransactionStatus.ABORTED,
txnOperated, "User Abort");
if (txnOperated) {
awaitTransactionState(logItem, transactionState);
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
b/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
index e7902076eea..509d9a87a5a 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
@@ -58,6 +58,7 @@ import org.apache.doris.common.jmockit.Deencapsulation;
import org.apache.doris.common.util.PropertyAnalyzer;
import org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob;
import
org.apache.doris.job.extensions.insert.streaming.StreamingTaskTxnCommitAttachment;
+import org.apache.doris.load.DeleteJob;
import org.apache.doris.load.routineload.RLTaskTxnCommitAttachment;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.SessionVariable;
@@ -75,6 +76,7 @@ import org.apache.doris.transaction.GlobalTransactionMgrIface;
import org.apache.doris.transaction.SubTransactionState;
import org.apache.doris.transaction.TabletCommitInfo;
import org.apache.doris.transaction.TransactionCommitFailedException;
+import org.apache.doris.transaction.TransactionNotFoundException;
import org.apache.doris.transaction.TransactionState;
import org.apache.doris.transaction.TransactionStatus;
import org.apache.doris.transaction.TxnStateChangeCallback;
@@ -719,6 +721,52 @@ public class CloudGlobalTransactionMgrTest {
}
}
+ @ParameterizedTest
+ @ValueSource(booleans = {false, true})
+ public void testAbortDeleteTransactionRemovesCallback(boolean
alreadyAborted) throws Exception {
+ long transactionId = 123535;
+ long jobId = 1002;
+ DeleteJob job = Mockito.spy(new DeleteJob(jobId, transactionId,
"delete_abort", Map.of(), null));
+ masterTransMgr.getCallbackFactory().addCallback(job);
+
+ MetaServiceProxy proxy = Mockito.mock(MetaServiceProxy.class);
+ try (MockedStatic<MetaServiceProxy> mocked =
Mockito.mockStatic(MetaServiceProxy.class)) {
+ mocked.when(MetaServiceProxy::getInstance).thenReturn(proxy);
+ TxnInfoPB txnInfo = buildTxnInfo(transactionId).toBuilder()
+ .setListenerId(jobId)
+ .setStatus(alreadyAborted ?
Cloud.TxnStatusPB.TXN_STATUS_ABORTED
+ : Cloud.TxnStatusPB.TXN_STATUS_PREPARED)
+ .build();
+
Mockito.when(proxy.getTxn(Mockito.any())).thenReturn(Cloud.GetTxnResponse.newBuilder()
+
.setStatus(Cloud.MetaServiceResponseStatus.newBuilder().setCode(MetaServiceCode.OK))
+ .setTxnInfo(txnInfo).build());
+ AbortTxnResponse.Builder response = AbortTxnResponse.newBuilder()
+ .setStatus(Cloud.MetaServiceResponseStatus.newBuilder()
+ .setCode(alreadyAborted ?
MetaServiceCode.TXN_ALREADY_ABORTED : MetaServiceCode.OK));
+ if (!alreadyAborted) {
+
response.setTxnInfo(txnInfo.toBuilder().setStatus(Cloud.TxnStatusPB.TXN_STATUS_ABORTED)
+ .setReason("User Cancelled"));
+ }
+
Mockito.when(proxy.abortTxn(Mockito.any())).thenReturn(response.build());
+
+ if (alreadyAborted) {
+ TransactionNotFoundException exception =
Assertions.assertThrows(TransactionNotFoundException.class,
+ () ->
masterTransMgr.abortTransaction(CatalogTestUtil.testDbId1, transactionId,
+ "User Cancelled"));
+ Assertions.assertTrue(exception.getMessage().contains("is
already aborted"), exception.getMessage());
+ } else {
+ masterTransMgr.abortTransaction(CatalogTestUtil.testDbId1,
transactionId, "User Cancelled");
+ }
+
+ Mockito.verify(job).beforeAborted(Mockito.any());
+ Mockito.verify(job).afterAborted(Mockito.any(),
Mockito.eq(!alreadyAborted), Mockito.anyString());
+
Assertions.assertNull(masterTransMgr.getCallbackFactory().getCallback(jobId),
+ "an aborted cloud DELETE must release its callback even
when MS aborted it first");
+ } finally {
+ masterTransMgr.getCallbackFactory().removeCallback(jobId);
+ }
+ }
+
@Test
public void testAbortRoutineLoadTransactionWithAttachment() throws
Exception {
long transactionId = 123534;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
b/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
index 3d41243955c..261413a12c9 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
@@ -17,6 +17,7 @@
package org.apache.doris.transaction;
+import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.catalog.BinlogConfig;
import org.apache.doris.catalog.CatalogTestUtil;
import org.apache.doris.catalog.ChangelogConfig;
@@ -47,6 +48,8 @@ import
org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob;
import
org.apache.doris.job.extensions.insert.streaming.StreamingTaskTxnCommitAttachment;
import org.apache.doris.job.offset.jdbc.JdbcOffset;
import org.apache.doris.job.offset.jdbc.JdbcSourceOffsetProvider;
+import org.apache.doris.load.DeleteJob;
+import org.apache.doris.load.routineload.kafka.KafkaRoutineLoadJob;
import org.apache.doris.meta.MetaContext;
import org.apache.doris.mtmv.ivm.IvmInfo;
import org.apache.doris.mysql.authenticate.TestLogAppender;
@@ -88,6 +91,7 @@ import java.util.Set;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.ReentrantReadWriteLock;
public class DatabaseTransactionMgrTest {
@@ -516,6 +520,205 @@ public class DatabaseTransactionMgrTest {
"unexpected message: " + e.getMessage());
}
+ private long beginAbortTestTransaction(String label, LoadJobSourceType
sourceType, boolean commit)
+ throws UserException {
+ Table table =
masterEnv.getInternalCatalog().getDbOrMetaException(CatalogTestUtil.testDbId1)
+ .getTableOrMetaException(CatalogTestUtil.testTableId1);
+ long txnId =
masterTransMgr.beginTransaction(CatalogTestUtil.testDbId1,
Lists.newArrayList(table.getId()),
+ label, transactionSource, sourceType,
Config.stream_load_default_timeout_second);
+ if (commit) {
+
masterTransMgr.commitTransactionWithoutLock(CatalogTestUtil.testDbId1,
Lists.newArrayList(table), txnId,
+
GlobalTransactionMgrTest.generateTabletCommitInfos(CatalogTestUtil.testTabletId1,
allBackends),
+ null);
+ }
+ return txnId;
+ }
+
+ private static void abortForTest(String mode, long txnId) throws
UserException {
+ DatabaseTransactionMgr dbTransMgr =
masterTransMgr.getDatabaseTransactionMgr(CatalogTestUtil.testDbId1);
+ if (mode.equals("2pc")) {
+ dbTransMgr.abortTransaction2PC(txnId);
+ } else {
+ dbTransMgr.abortTransaction(txnId, "abort for test", null);
+ }
+ }
+
+ // Registers a real routine load job as the callback of the transaction,
whose beforeAborted() takes the job's
+ // write lock and whose afterAborted() releases it. Records whether
afterAborted() ran under the txn monitor.
+ private static KafkaRoutineLoadJob
addRoutineLoadJobCallback(TransactionState state,
+ AtomicBoolean afterAbortedHoldsMonitor) throws UserException {
+ KafkaRoutineLoadJob job = Mockito.spy(new
KafkaRoutineLoadJob(state.getTransactionId(), "abort_test_job",
+ CatalogTestUtil.testDbId1, CatalogTestUtil.testTableId1,
"host:port", "topic", UserIdentity.ADMIN));
+ Mockito.doAnswer(invocation -> {
+ afterAbortedHoldsMonitor.set(Thread.holdsLock(state));
+ return invocation.callRealMethod();
+ }).when(job).afterAborted(Mockito.any(), Mockito.anyBoolean(),
Mockito.any());
+ masterTransMgr.getCallbackFactory().addCallback(job);
+ state.setCallbackId(job.getId());
+ return job;
+ }
+
+ @ParameterizedTest
+ @CsvSource({"normal,false", "normal,true", "2pc,false", "2pc,true"})
+ public void testRejectedAbortReleasesRoutineLoadJobLock(String mode,
boolean logOutsideLock) throws Exception {
+ boolean originalLogOutsideLock = Config.enable_txn_log_outside_lock;
+ ReentrantReadWriteLock jobLock = null;
+ try {
+ Config.enable_txn_log_outside_lock = logOutsideLock;
+ FakeEnv.setEnv(masterEnv);
+ long txnId = beginAbortTestTransaction("rejected_abort_" + mode +
"_" + logOutsideLock,
+ LoadJobSourceType.ROUTINE_LOAD_TASK, true);
+ TransactionState state =
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+ Assertions.assertEquals(TransactionStatus.COMMITTED,
state.getTransactionStatus());
+ AtomicBoolean afterAbortedHoldsMonitor = new AtomicBoolean(true);
+ KafkaRoutineLoadJob job = addRoutineLoadJobCallback(state,
afterAbortedHoldsMonitor);
+ jobLock = Deencapsulation.getField(job, "lock");
+ Mockito.clearInvocations(masterEnv.getEditLog());
+
+ // A late abort, such as a BE rollback after its commit RPC timed
out, is rejected.
+ UserException e = Assertions.assertThrows(UserException.class, ()
-> abortForTest(mode, txnId));
+ Assertions.assertTrue(e.getMessage().contains("is already
COMMITTED, could not abort"), e.getMessage());
+ Assertions.assertEquals(TransactionStatus.COMMITTED,
state.getTransactionStatus());
+ Mockito.verifyNoInteractions(masterEnv.getEditLog());
+
+ // The publish thread must still finish the transaction:
afterVisible() takes the job's write lock.
+ Map<String, Map<Long, Long>> successTablets = new HashMap<>();
+ setSuccessTablet(successTablets, allBackends, txnId,
CatalogTestUtil.testTabletId1,
+ CatalogTestUtil.testStartVersion + 2);
+ setTransactionFinishPublish(state, allBackends, successTablets);
+ FutureTask<Void> publishTask = new FutureTask<>(() -> {
+ try (FakeEnv threadEnv = new FakeEnv()) {
+
masterTransMgr.finishTransaction(CatalogTestUtil.testDbId1, txnId,
Maps.newHashMap(),
+ Maps.newHashMap());
+ return null;
+ }
+ });
+ new Thread(publishTask, "publish-after-rejected-abort").start();
+ Assertions.assertDoesNotThrow(() -> publishTask.get(10,
TimeUnit.SECONDS),
+ "publishing the committed transaction waits for the
routine load job lock");
+ Assertions.assertEquals(TransactionStatus.VISIBLE,
state.getTransactionStatus());
+
+ Mockito.verify(job).beforeAborted(state);
+ Mockito.verify(job).afterAborted(Mockito.eq(state),
Mockito.eq(false), Mockito.any());
+ Assertions.assertFalse(afterAbortedHoldsMonitor.get(),
"afterAborted() ran under the transaction monitor");
+ Mockito.verify(job).afterVisible(state, true);
+ Assertions.assertFalse(jobLock.isWriteLocked(), "the routine load
job lock is still held");
+ } finally {
+ Config.enable_txn_log_outside_lock = originalLogOutsideLock;
+ // Release a lock leaked on this thread so that a blocked publish
thread can exit.
+ while (jobLock != null && jobLock.isWriteLockedByCurrentThread()) {
+ jobLock.writeLock().unlock();
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @CsvSource({"normal,false,false", "normal,true,false", "2pc,false,false",
"2pc,true,false",
+ "normal,false,true", "normal,true,true", "2pc,false,true",
"2pc,true,true"})
+ public void testAbortReleasesRoutineLoadJobLock(String mode, boolean
logOutsideLock, boolean failEditLog)
+ throws Exception {
+ boolean originalLogOutsideLock = Config.enable_txn_log_outside_lock;
+ ReentrantReadWriteLock jobLock = null;
+ try {
+ Config.enable_txn_log_outside_lock = logOutsideLock;
+ FakeEnv.setEnv(masterEnv);
+ long txnId = beginAbortTestTransaction("abort_" + mode + "_" +
logOutsideLock + "_" + failEditLog,
+ LoadJobSourceType.ROUTINE_LOAD_TASK, false);
+ TransactionState state =
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+ AtomicBoolean afterAbortedHoldsMonitor = new AtomicBoolean(true);
+ KafkaRoutineLoadJob job = addRoutineLoadJobCallback(state,
afterAbortedHoldsMonitor);
+ jobLock = Deencapsulation.getField(job, "lock");
+
+ if (failEditLog) {
+ // What EditLog throws when it has no edit log stream; the
abort is already done in memory.
+ Error editLogFailure = new Error("Fatal Error : no editLog
stream");
+ Mockito.doThrow(editLogFailure).when(masterEnv.getEditLog())
+
.logInsertTransactionState(Mockito.any(TransactionState.class));
+ Mockito.doThrow(editLogFailure).when(masterEnv.getEditLog())
+ .submitEdit(Mockito.anyShort(), Mockito.any());
+ Error thrown = Assertions.assertThrows(Error.class, () ->
abortForTest(mode, txnId));
+ Assertions.assertSame(editLogFailure, thrown);
+ } else {
+ abortForTest(mode, txnId);
+ }
+
+ Assertions.assertFalse(jobLock.isWriteLocked(), "the routine load
job lock is still held");
+ Assertions.assertEquals(TransactionStatus.ABORTED,
state.getTransactionStatus());
+ Mockito.verify(job).beforeAborted(state);
+ // The transaction is aborted in memory before the edit log write,
so txnOperated is true either way.
+ Mockito.verify(job).afterAborted(Mockito.eq(state),
Mockito.eq(true), Mockito.any());
+ Assertions.assertFalse(afterAbortedHoldsMonitor.get(),
"afterAborted() ran under the transaction monitor");
+ } finally {
+ Config.enable_txn_log_outside_lock = originalLogOutsideLock;
+ while (jobLock != null && jobLock.isWriteLockedByCurrentThread()) {
+ jobLock.writeLock().unlock();
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"normal", "2pc"})
+ public void testAbortSkipsAfterCallbackWhenBeforeCallbackFails(String
mode) throws Exception {
+ FakeEnv.setEnv(masterEnv);
+ long txnId = beginAbortTestTransaction("before_callback_fails_" +
mode, LoadJobSourceType.FRONTEND, false);
+ TransactionState state =
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+ TxnStateChangeCallback callback =
Mockito.mock(TxnStateChangeCallback.class);
+ Mockito.when(callback.getId()).thenReturn(txnId);
+ TransactionException beforeFailure = new TransactionException("before
callback failed");
+ Mockito.doThrow(beforeFailure).when(callback).beforeAborted(state);
+ masterTransMgr.getCallbackFactory().addCallback(callback);
+ state.setCallbackId(txnId);
+
+ TransactionException thrown =
Assertions.assertThrows(TransactionException.class,
+ () -> abortForTest(mode, txnId));
+ Assertions.assertSame(beforeFailure, thrown);
+ Assertions.assertEquals(TransactionStatus.PREPARE,
state.getTransactionStatus());
+ Mockito.verify(callback, Mockito.never()).afterAborted(Mockito.any(),
Mockito.anyBoolean(), Mockito.any());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"normal", "2pc"})
+ public void
testRejectedAbortKeepsAbortFailureWhenAfterCallbackFails(String mode) throws
Exception {
+ FakeEnv.setEnv(masterEnv);
+ long txnId = beginAbortTestTransaction("after_callback_fails_" + mode,
LoadJobSourceType.FRONTEND, true);
+ TransactionState state =
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+ TxnStateChangeCallback callback =
Mockito.mock(TxnStateChangeCallback.class);
+ Mockito.when(callback.getId()).thenReturn(txnId);
+ Mockito.doThrow(new RuntimeException("after callback
failed")).when(callback)
+ .afterAborted(Mockito.any(), Mockito.anyBoolean(),
Mockito.any());
+ masterTransMgr.getCallbackFactory().addCallback(callback);
+ state.setCallbackId(txnId);
+
+ UserException e = Assertions.assertThrows(UserException.class, () ->
abortForTest(mode, txnId));
+ Assertions.assertTrue(e.getMessage().contains("is already COMMITTED,
could not abort"), e.getMessage());
+ Assertions.assertEquals(TransactionStatus.COMMITTED,
state.getTransactionStatus());
+ Mockito.verify(callback).afterAborted(Mockito.eq(state),
Mockito.eq(false), Mockito.any());
+ }
+
+ @Test
+ public void testRejectedAbortKeepsDeleteJobCallback() throws Exception {
+ FakeEnv.setEnv(masterEnv);
+ TxnStateCallbackFactory callbackFactory =
masterTransMgr.getCallbackFactory();
+ // DeleteJob.cancel() aborts the transaction; a committed delete still
needs afterVisible() to finish.
+ long committedTxnId = beginAbortTestTransaction("delete_committed",
LoadJobSourceType.FRONTEND, true);
+ DeleteJob committedJob = new DeleteJob(committedTxnId, committedTxnId,
"delete_committed",
+ Maps.newHashMap(), null);
+ callbackFactory.addCallback(committedJob);
+ masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1,
committedTxnId)
+ .setCallbackId(committedJob.getId());
+ Assertions.assertThrows(UserException.class, () ->
abortForTest("normal", committedTxnId));
+ Assertions.assertSame(committedJob,
callbackFactory.getCallback(committedJob.getId()));
+
+ long preparedTxnId = beginAbortTestTransaction("delete_prepared",
LoadJobSourceType.FRONTEND, false);
+ DeleteJob abortedJob = new DeleteJob(preparedTxnId, preparedTxnId,
"delete_prepared", Maps.newHashMap(),
+ null);
+ callbackFactory.addCallback(abortedJob);
+ masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1,
preparedTxnId)
+ .setCallbackId(abortedJob.getId());
+ abortForTest("normal", preparedTxnId);
+ Assertions.assertNull(callbackFactory.getCallback(abortedJob.getId()));
+ }
+
@Test
public void
testUpdateCatalogAfterCommittedAdvancesIvmSequencePrefixForNormalCommitAndReplay()
throws Exception {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]