JackieTien97 commented on code in PR #18909:
URL: https://github.com/apache/iotdb/pull/18909#discussion_r4322366682


##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java:
##########
@@ -730,12 +751,101 @@ private LockStateResult<Env> executeRootStackRollback(
       }
     }
 
+    if (!proceduresKeepingTheirStates.isEmpty()) {
+      subprocStack.removeIf(proceduresKeepingTheirStates::contains);
+      // The children go first: releasing a child's lock writes its record 
back, which has to be
+      // covered by the root's cleanup. The root is finished last so that no 
write of a child
+      // follows the write which persists the root as finished.
+      for (Procedure<Env> procedure : proceduresKeepingTheirStates) {
+        if (procedure.hasParent()) {
+          finishWithoutRollback(procedure);
+        }
+      }
+      for (Procedure<Env> procedure : proceduresKeepingTheirStates) {
+        if (!procedure.hasParent()) {
+          finishWithoutRollback(procedure);
+        }
+      }
+    }
+
     LOG.info(
         ProcedureMessages.ROLLED_BACK_TIME_DURATION_IS, rootProcedure, 
rootProcedure.elapsedTime());
     rootProcedureCleanup(rootProcedure);
     return new LockStateResult<>(ProcedureLockState.LOCK_ACQUIRED, 
rootProcedure);
   }
 
+  /**
+   * Finishes a procedure without undoing any of the states it has executed. 
The failure is still
+   * reported, as the operation the procedure was part of did not complete; 
what is dropped is only
+   * the bookkeeping of the rollback, so that neither this rollback nor the 
one a new leader
+   * recovers touches this procedure again.
+   */
+  private void finishWithoutRollback(final Procedure<Env> procedure) {
+    // The unlock persists the procedure through doReleaseLock(). It is done 
first, while the
+    // procedure is neither marked finished nor stripped of its state history, 
so that a snapshot
+    // taken right after this write is still a state recovery resumes. Forcing 
the release matters
+    // because the procedure is not finished yet, so releaseLock() would 
otherwise keep the lock of
+    // procedures which hold it across steps.
+    releaseLockIfHeld(procedure);
+    if (procedure.hasParent()) {
+      // Mark the procedure finished before dropping its state history, so 
that a worker which
+      // picks it up concurrently returns at the isFinished() check instead of 
rolling it back.
+      if (!procedure.isSuccess()) {
+        procedure.setState(ProcedureState.ROLLEDBACK);
+      }
+      procedure.abandonRollback();
+      try {
+        store.delete(procedure.getProcId());
+        // do not remove this procedure when exception occurred
+        procedures.remove(procedure.getProcId());
+      } catch (final Exception e) {
+        LOG.warn(ProcedureMessages.FAILED_TO_DELETE_PROCEDURE_ON_ROLLBACK, e);
+      }
+    } else {
+      // A child which never executed has no rollback step and is not visited 
above, so its record
+      // goes with the ones of the children which were.
+      // Every child record has to be gone before the root may be persisted as 
finished: a snapshot
+      // holding a finished root beside a child record makes recover() fail on 
that child. Deleting
+      // is idempotent, so it is retried until durable.
+      final long[] childProcIds = 
rollbackStack.get(procedure.getProcId()).getSubprocedureIds();
+      for (final long childProcId : childProcIds) {
+        RetryUtils.executeWithEndlessBackoffRetry(
+            () -> store.delete(childProcId),
+            
ProcedureMessages.LOG_DELETE_CHILD_PROCEDURES_WITHOUT_ROLLBACK_6E62D1C1);
+      }

Review Comment:
   [P1] Abort root finalization when the child-cleanup retry is interrupted
   
   `RetryUtils.executeWithEndlessBackoffRetry()` returns normally when its 
backoff sleep is interrupted: it restores the interrupt flag but neither 
rethrows nor reports that the operation did not succeed. Consequently, 
returning from this call does not guarantee that `store.delete(childProcId)` 
completed. If cleanup fails and the worker is interrupted during retry, this 
method still proceeds to mark the root `ROLLEDBACK`, discard its history, and 
attempt to persist its terminal state. If that update is accepted, a child 
record remains beside the completed root.
   
   I reproduced this with the clean build of this revision and real 
`RemoveDataNodesProcedure` / `RegionMigrateProcedure` serialization: inject one 
child-deletion failure, then interrupt the thread during the retry's backoff. 
The final snapshot retains the `ROLLEDBACK` child, and initializing a new 
executor from it throws `UnsupportedOperationException("Unexpected state")`. 
The same tests pass when the transient deletion failure is allowed to retry 
without interruption.
   
   Please explicitly distinguish successful cleanup from interrupted cleanup 
and propagate cancellation out of the entire rollback path, so neither the 
terminal-state write nor `rootProcedureCleanup()` runs after incomplete child 
deletion. Simply returning from `finishWithoutRollback()` would still allow its 
caller to perform root cleanup.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to