qqeasonchen opened a new pull request, #5345: URL: https://github.com/apache/eventmesh/pull/5345
<!-- Part of #5296 (Architecture Review, "New review questions" 2026-09-07, Q3). Closes #5339. --> ## State store durability tests (issue #5339) Q3 asks: does each state-store contract have the intended durable implementation, and are restart, multi-instance, fencing, and cross-store failure scenarios covered by executable tests rather than documentation only? The unified state control plane landed in #5301 Sub-PR A / B / C / D; the production implementations are correct, but the acceptance tests for the durability scenarios were not yet on disk. This PR fills that gap. ### 1. `eventmesh-runtime/src/test/java/.../StateStoreDurabilityTest.java` (new) A single JUnit 5 test class with six `@Nested` scenarios — one per item in the issue #5339 acceptance list: | # | Scenario | Store | What it asserts | | --- | --- | --- | --- | | 1 | `OffsetStoreRestart` | `OffsetStore` (RocksDB) | Persist offset 100, kill runtime, restart, read = 100. Monotonic non-decreasing invariant (issue #5289) is enforced across restarts. | | 2 | `SubscriptionStoreMultiInstance` | `SubscriptionStore` (Meta + local cache) | Two `ClusterSubscriptionStore` instances share one `InMemoryMetaStore`; a 100-iteration concurrent register / unregister storm converges on the same view (last-write-wins + watch prefix). The `removeIsObservedByAllInstances` test confirms the no-stale-view property. | | 3 | `SessionStoreFencing` | `SessionStore` (Meta) | Instance A registers agent X, unregisters, instance B takes over. A's subsequent heartbeat on its stale handle is rejected — and the deterministic `agent-Y-never-registered` case pins the fencing property. | | 4 | `DeadLetterStoreRestart` | `DeadLetterStore` (Meta CAS) | Three DLQ entries, close the wrapper, re-open over the same Meta — the ledger is preserved. Idempotent re-record and post-partition heal are also covered. | | 5 | `TaskStoreMetaFailure` | `TaskStore` (Meta CAS + epoch) | `MetaPartitionSwitch` around `MetaBackedTaskStore`. `createTask` and `updateStatus` both surface `MetaPartitionException` rather than silently dropping the task. Heal + retry succeeds with no half-state. | | 6 | `DeliveryStateStoreKillMinusNine` | `DeliveryStateStore` (RocksDB) | Drop the `RocksDBDeliveryStateStore` without graceful close; re-open at the same path; every persisted delivery is readable (count + iterate + per-id get). The `nextAttemptAtMs` field is preserved so retry timing resumes on the fresh JVM. | All six tests are pure JUnit (no Testcontainers); they run in the default `test` task and exercise the production implementations directly. The in-memory test impls they consume are already on the runtime test classpath (`InMemoryMetaStore`, `MetaPartitionSwitch`). ### 2. `docs/state-store-failure-matrix.md` (new) Canonical per-backend failure matrix (RocksDB / Meta) documenting the operator-visible signal and recovery path for every failure mode: * **RocksDB backend** (OffsetStore, DeliveryStateStore): local disk full, process kill -9, corrupted file, graceful restart. * **Meta backend** (SubscriptionStore, SessionStore, DeadLetterStore, TaskStore): Meta unreachable, cluster split-brain, stale data (clock skew), Meta restart, in-process cache divergence. Section 3 maps every issue #5339 acceptance scenario to the test that exercises it, so the evidence table in #5337 can reference this doc directly. ### Acceptance check for #5339 - [x] For each of the 6 stores above, a JUnit test exists in `eventmesh-runtime` and passes in CI. - [x] The test for each scenario is referenced in the section 3 evidence table in `docs/state-store-failure-matrix.md`. - [x] No production-code store is backed by an `InMemory*` implementation; the in-memory implementations live in `state/fault/` (test-only). - [x] `docs/state-store-failure-matrix.md` exists in `docs/`. ### Out of scope * Testcontainers E2E for A2A (`#5340`). * `MetaAgentCardRegistry` and `TaskExpirer` reaper (`#5340`). * The architecture-guard verification PR (`#5342`). * The closure-criteria evidence table (`#5337`). Closes #5339 Part of #5296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
