Caideyipi commented on code in PR #18909:
URL: https://github.com/apache/iotdb/pull/18909#discussion_r4311387960
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java:
##########
@@ -730,12 +746,60 @@ private LockStateResult<Env> executeRootStackRollback(
}
}
+ if (!proceduresKeepingTheirStates.isEmpty()) {
+ subprocStack.removeIf(proceduresKeepingTheirStates::contains);
+ proceduresKeepingTheirStates.forEach(this::finishWithoutRollback);
+ }
+
LOG.info(
ProcedureMessages.ROLLED_BACK_TIME_DURATION_IS, rootProcedure,
rootProcedure.elapsedTime());
rootProcedureCleanup(rootProcedure);
return new LockStateResult<>(ProcedureLockState.LOCK_ACQUIRED,
rootProcedure);
}
+ /**
+ * Finishes a procedure without undoing any of the states it has executed.
The failure is still
+ * reported, as the operation the procedure was part of did not complete;
what is dropped is only
+ * the bookkeeping of the rollback, so that neither this rollback nor the
one a new leader
+ * recovers touches this procedure again.
+ */
+ private void finishWithoutRollback(final Procedure<Env> procedure) {
+ procedure.abandonRollback();
+ if (!procedure.isSuccess()) {
+ procedure.setState(ProcedureState.ROLLEDBACK);
+ }
+ if (procedure.hasParent()) {
+ try {
+ store.delete(procedure.getProcId());
+ // do not remove this procedure when exception occurred
+ procedures.remove(procedure.getProcId());
+ } catch (final Exception e) {
+ LOG.warn(ProcedureMessages.FAILED_TO_DELETE_PROCEDURE_ON_ROLLBACK, e);
+ }
+ } else {
+ // A child which never executed has no rollback step and is not visited
above, so its record
+ // goes with the ones of the children which were.
+ final long[] childProcIds =
rollbackStack.get(procedure.getProcId()).getSubprocedureIds();
+ if (childProcIds.length > 0) {
+ try {
+ store.delete(childProcIds);
+ } catch (final Exception e) {
+ LOG.warn(ProcedureMessages.FAILED_TO_DELETE_PROCEDURE_ON_ROLLBACK,
e);
+ }
+ }
+ // The stack indexes and the states just dropped are persisted here, so
that the rollback
+ // recovered by a new leader leaves this procedure out. Endless retry
since this step is not
+ // idempotent.
+ RetryUtils.executeWithEndlessBackoffRetry(
+ () -> store.update(procedure), "finish procedure without rollback");
Review Comment:
[P3] Localize the new retry operation name
`RetryUtils.executeWithEndlessBackoffRetry()` includes `operationName` in
its retry, interruption, and eventual-success logs, so this new string is
user-visible. Keeping `"finish procedure without rollback"` inline leaves that
part of the message in English in the Chinese build. Please extract it into a
message constant with matching entries in the en/zh `ProcedureMessages` classes.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java:
##########
@@ -708,6 +720,10 @@ private LockStateResult<Env> executeRootStackRollback(
cleanupAfterRollback(procedure);
continue;
}
+ if (proceduresKeepingTheirStates.contains(procedure)) {
+ // Finished below, once the rest of the stack has been rolled back.
+ continue;
Review Comment:
[P1] Release restored locks when skipping rollback
On leader recovery, `Procedure.restoreLock()` reacquires the underlying
lock, but leaves `hasLock()` false: `doAcquireLock()` normally transfers
`lockedWhenLoading` into `locked`. This `continue` skips that bookkeeping, and
`finishWithoutRollback()` does not release the lock either. Consequently,
`rootProcedureCleanup()` sees `hasLock() == false` and leaves the restored lock
held by a procedure that is already finished.
I reproduced this with a persisted, failed `AddConfigNodeProcedure` at
`REGISTER_SUCCESS` with its lock flag set. After recovery, the procedure
finishes as `ROLLEDBACK`, but the real `LockQueue` still names it as the global
`nodeLock` owner, so subsequent node-management procedures sharing that lock
cannot proceed. The same test releases the lock on the base revision. Please
preserve the lock bookkeeping and cleanup on this path while skipping the
rollback callbacks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]