d-wang-commit opened a new pull request, #19007:
URL: https://github.com/apache/iotdb/pull/19007
## Description
### Persist the parent's progress together with newly submitted children
When a Procedure returns new children,
`ProcedureExecutor.initializeChildren` advances the parent's execution state to
WAITING. The existing persistence path writes only the children before
scheduling them. A later lock release can persist the parent, but a crash
before that write leaves an old RUNNABLE parent alongside its already persisted
children. Procedures that hold their lock across child execution, such as
`RemoveDataNodesProcedure`, cannot rely on that later lock-release write either.
On recovery, the executor can schedule both the stale parent and its
unfinished children. Re-executing the parent's submission step creates
duplicate children with different procedure IDs. For region migration,
competing add-peer procedures can advance and roll back the same replica
independently.
Persist the parent's next state and all new children before scheduling any
child. `ConfigProcedureStore` writes this batch as one `UpdateProceduresPlan`
consensus entry, checks its returned status, and lets the executor retry
failures with the existing backoff helper. An interrupted persistence retry
must not fall through to scheduling children.
### Preserve recovery and snapshot consistency
Each procedure in the batch is length-delimited and uses the existing
single-procedure encoding, so optional trailing fields cannot consume the next
record. Apply the batch under the same `ProcedureInfo` monitor used by recovery
reads and snapshot creation, preventing either operation from observing a
partially applied batch. Existing single-procedure entries and snapshot file
formats remain readable.
### Tests
- Added `ProcedureChildPersistenceTest`: restart after one or several
children are persisted, retry a failed batch before executing children, and
stop during a failing retry without starting children. It exercises the real
executor and serializers with a recording store and a controlled stop at the
persistence boundary.
- Negative control on the existing upstream executor: both restart tests
failed because the saved parent remained RUNNABLE instead of WAITING. The test
parent uses the existing default lock behavior; no production fault hook or
downstream lock override is needed.
- Added `UpdateProceduresPlanTest`: batch serialization and snapshot
recovery, preservation of the old single-entry reader, one consensus write per
batch, and rejection-status propagation.
- Passed 142 UTs across `ProcedureChildPersistenceTest`,
`UpdateProceduresPlanTest`, `TestProcedureExecutor`, `STMProcedureTest`,
`StateMachineProcedureTest`, `ProcedureInfoTest`,
`ConfigPhysicalPlanSerDeTest`, and `ConfigPhysicalPlanTypeTest`.
- IT: none added or modified in this PR. Passed
`IoTDBRegionMigrateConfigNodeCrashForRatisIT#cnCrashDuringCreatePeerTest` and
`IoTDBRegionMigrateConfigNodeCrashIoTV1IT#cnCrashDuringCreatePeerTest`, using
the unmodified upstream IT assertions on this independent branch.
- The UT/IT run used `mvn verify -pl integration-test -am
-PDailyIT,with-integration-tests` with the above selections and
no-specified-test flags for upstream reactor modules. Spotless passed.
- Full reactor `mvn package -am -DskipTests` and `mvn package -am
-DskipTests -Pwith-zh-locale` both passed.
### Side effects and risks
This introduces ConfigNode plan type 502. New readers retain support for old
entries, but old ConfigNodes cannot decode the new batch type; mixed-version
rolling upgrade and downgrade are not established by these tests. A batch entry
grows with the parent and child records, and snapshot creation now excludes
batch updates for its duration. Large child batches and slow snapshot disks
were not stress-tested.
The change prevents duplicate child submission caused by mismatched
persisted parent/child states. It does not make external DataNode actions
exactly-once, repair existing inconsistent procedure records, or replace
individual procedures' retry and rollback logic. Persistent store failures keep
the children unscheduled until the batch succeeds or the executor is
interrupted.
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are
corner cases and error
conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes,
inheritance, composition,
design patterns)
- Method organization and design (how the logic is split between methods,
parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of
emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative
name) for every design
(or naming) decision point and compare the alternatives with the designs
that you've implemented
(or the names you've chosen) to highlight the advantages of the chosen
designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in
this PR elsewhere
(e. g. a "Proposal" issue, any other issue, or a thread in the development
mailing list),
link to that discussion from this PR description and explain what have
changed in your final design
compared to your original proposal or the consensus version in the end of
the discussion.
If something hasn't changed since the original discussion, you can omit a
detailed discussion of
those aspects of the design here, perhaps apart from brief mentioning for
the sake of readability
of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small
changes. -->
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read and write
- [x] added comments explaining the persistence ordering and record
boundaries.
- [x] added unit tests covering the new persistence and recovery paths.
- [x] been tested in a test IoTDB cluster.
<!-- Check the items by putting "x" in the brackets for the done things. Not
all of these items
apply to every PR. Remove the items which are not done or not relevant to
the PR. None of the items
from the checklist above are strictly necessary, but it would be very
helpful if you at least
self-review the PR. -->
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `ProcedureExecutor`: persist the parent and new children before
scheduling, retry store failures, and stop after interruption.
- `ConfigProcedureStore` / `IProcedureStore`: use and document atomic batch
persistence.
- `UpdateProceduresPlan` / `ConfigPhysicalPlan` / `ConfigPhysicalPlanType` /
`ConfigPlanExecutor`: encode, decode, and apply the batch consensus entry.
- `ProcedureInfo`: exclude recovery reads and snapshots from partially
applied batches.
- `ProcedureChildPersistenceTest` / `UpdateProceduresPlanTest`: exercise
recovery, failures, serialization, and consensus submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]