JackieTien97 commented on code in PR #18909:
URL: https://github.com/apache/iotdb/pull/18909#discussion_r4322366682
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java:
##########
@@ -730,12 +751,101 @@ private LockStateResult<Env> executeRootStackRollback(
}
}
+ if (!proceduresKeepingTheirStates.isEmpty()) {
+ subprocStack.removeIf(proceduresKeepingTheirStates::contains);
+ // The children go first: releasing a child's lock writes its record
back, which has to be
+ // covered by the root's cleanup. The root is finished last so that no
write of a child
+ // follows the write which persists the root as finished.
+ for (Procedure<Env> procedure : proceduresKeepingTheirStates) {
+ if (procedure.hasParent()) {
+ finishWithoutRollback(procedure);
+ }
+ }
+ for (Procedure<Env> procedure : proceduresKeepingTheirStates) {
+ if (!procedure.hasParent()) {
+ finishWithoutRollback(procedure);
+ }
+ }
+ }
+
LOG.info(
ProcedureMessages.ROLLED_BACK_TIME_DURATION_IS, rootProcedure,
rootProcedure.elapsedTime());
rootProcedureCleanup(rootProcedure);
return new LockStateResult<>(ProcedureLockState.LOCK_ACQUIRED,
rootProcedure);
}
+ /**
+ * Finishes a procedure without undoing any of the states it has executed.
The failure is still
+ * reported, as the operation the procedure was part of did not complete;
what is dropped is only
+ * the bookkeeping of the rollback, so that neither this rollback nor the
one a new leader
+ * recovers touches this procedure again.
+ */
+ private void finishWithoutRollback(final Procedure<Env> procedure) {
+ // The unlock persists the procedure through doReleaseLock(). It is done
first, while the
+ // procedure is neither marked finished nor stripped of its state history,
so that a snapshot
+ // taken right after this write is still a state recovery resumes. Forcing
the release matters
+ // because the procedure is not finished yet, so releaseLock() would
otherwise keep the lock of
+ // procedures which hold it across steps.
+ releaseLockIfHeld(procedure);
+ if (procedure.hasParent()) {
+ // Mark the procedure finished before dropping its state history, so
that a worker which
+ // picks it up concurrently returns at the isFinished() check instead of
rolling it back.
+ if (!procedure.isSuccess()) {
+ procedure.setState(ProcedureState.ROLLEDBACK);
+ }
+ procedure.abandonRollback();
+ try {
+ store.delete(procedure.getProcId());
+ // do not remove this procedure when exception occurred
+ procedures.remove(procedure.getProcId());
+ } catch (final Exception e) {
+ LOG.warn(ProcedureMessages.FAILED_TO_DELETE_PROCEDURE_ON_ROLLBACK, e);
+ }
+ } else {
+ // A child which never executed has no rollback step and is not visited
above, so its record
+ // goes with the ones of the children which were.
+ // Every child record has to be gone before the root may be persisted as
finished: a snapshot
+ // holding a finished root beside a child record makes recover() fail on
that child. Deleting
+ // is idempotent, so it is retried until durable.
+ final long[] childProcIds =
rollbackStack.get(procedure.getProcId()).getSubprocedureIds();
+ for (final long childProcId : childProcIds) {
+ RetryUtils.executeWithEndlessBackoffRetry(
+ () -> store.delete(childProcId),
+
ProcedureMessages.LOG_DELETE_CHILD_PROCEDURES_WITHOUT_ROLLBACK_6E62D1C1);
+ }
Review Comment:
[P1] Abort root finalization when the child-cleanup retry is interrupted
`RetryUtils.executeWithEndlessBackoffRetry()` returns normally when its
backoff sleep is interrupted: it restores the interrupt flag but neither
rethrows nor reports that the operation did not succeed. Consequently,
returning from this call does not guarantee that `store.delete(childProcId)`
completed. If cleanup fails and the worker is interrupted during retry, this
method still proceeds to mark the root `ROLLEDBACK`, discard its history, and
attempt to persist its terminal state. If that update is accepted, a child
record remains beside the completed root.
I reproduced this with the clean build of this revision and real
`RemoveDataNodesProcedure` / `RegionMigrateProcedure` serialization: inject one
child-deletion failure, then interrupt the thread during the retry's backoff.
The final snapshot retains the `ROLLEDBACK` child, and initializing a new
executor from it throws `UnsupportedOperationException("Unexpected state")`.
The same tests pass when the transient deletion failure is allowed to retry
without interruption.
Please explicitly distinguish successful cleanup from interrupted cleanup
and propagate cancellation out of the entire rollback path, so neither the
terminal-state write nor `rootProcedureCleanup()` runs after incomplete child
deletion. Simply returning from `finishWithoutRollback()` would still allow its
caller to perform root cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]