[
https://issues.apache.org/jira/browse/HDDS-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098364#comment-18098364
]
Ivan Andika commented on HDDS-15501:
------------------------------------
Another concern is that since the spec is not coupled to the implementation,
the spec can become stale if we change any of the implementation. We need a way
to ensure the spec remains in-sync with the latest implementation. One way is
to have a workflow that will trigger job to update the spec (similar to how the
configuration change in ozone-main can raise update the configuration in
ozone-site).
> Distributed System Testing in Ozone
> -----------------------------------
>
> Key: HDDS-15501
> URL: https://issues.apache.org/jira/browse/HDDS-15501
> Project: Apache Ozone
> Issue Type: Epic
> Components: test
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, we only test Ozone using the traditional UT, IT, Acceptance Tests.
> We had a MiniOzoneChaosCluster (fault injection testing), but it seems
> unused. Additionally, we also have byteman injection testing but it seems
> that it is a part of acceptance tests and therefore the testing feedback loop
> is not that tight. I propose to introduce a distributed system testing and
> proofs system so that we can have the Ozone spec as the shared mental model.
> Some of the regressions for issues like breaking majority commit contract
> (HDDS-15052) or handling disk failures (HDDS-12564, HDDS-12151) is not
> detected since we don't have the spec as the source of truth. Additionally
> sometimes simply we use our intuitions to guide our implementation and fixes
> which can cause regressions (for example, a lot of ReplicationManager fixes
> are only done only when there is an issue in productions).
> This is a parent task for the effort to introduce distributed system testing
> and proofs to test the correctness of Ozone implementation. For example,
> * Partial write commit and all commit semantics
> * Container state transitions (under failures)
> * Replication manager & container balancer correctness
> ** Replication manager should not cause infinite replications, it should end
> in a steady state
> * Container state and replica management
> ** How to appease eventually consistent container replica management
> (through heartbeat) and strongly consistent container state (through Ratis)
> * End to end block deletion guarantee
> ** Ensure that blocks are deleted and not orphaned
> * Ensure that container replica numOfKeys and numOfBytes is accurate
> * OM bucket quota accuracy issue: Ensure that OM bucket quota should not
> become negative
> * OM and SCM linearizability guarantee with the table cache and (double)
> transacstion buffer mechanisms
> * Container reconciliation guarantee
> * SCM distributed sequence ID generator correctness
> ** ID generated by ID generator is globally unique in presence of failures
> Distributed system testing tools:
> - Jepsen, Ellen, Maelstorm
> - Fray
> - Hypothesis (Hegel)
> - Antithesis (paid)
> Distributed system proofs:
> - TLA+
> -- Specula ([https://github.com/specula-org/Specula]) : AI generated TLA+
> already used to find some Raft bugs
> - Lean4
> - P framework
> Real systems
> - 3FS ([https://github.com/deepseek-ai/3FS/tree/main/specs]) - uses P
> framework
> - AWS S3
> --
> [https://cacm.acm.org/practice/systems-correctness-practices-at-amazon-web-services/]
> -- [https://p-org.github.io/P/casestudies/#case-studies]
> -- https://dl.acm.org/doi/10.1145/3477132.3483540
> - etcd robustness test
> ([https://github.com/etcd-io/etcd/tree/main/tests/robustness]) - Uses
> antithesis (among other things)
> I prefer if we can start with P framework since some storage systems already
> used it and it seems it already had some JUnit integration.
> In the future, we can support Deterministic Simulation Testing
> ([https://antithesis.com/docs/resources/deterministic_simulation_testing]).
> However, this requires a lot of efforts and very invasive since we need to
> make all Ozone implementations to be testable by the simulation framework so
> it's not going to happen anytime soon. Some of the deterministic simulation
> testing framework is Madsim (Rust), Turmoil (Rust), BUGGIFY (C++,
> FoundationDB), Tickloom (Java, [https://github.com/unmeshjoshi/tickloom]),
> Vortex (Zig, used by TigerBeetle). We can start taking look at Tickloom for
> testing in Java.
> Having a real industry-recognized spec helps to instil confidence in Ozone
> robustness. More importantly, distributed system testing allows us to have
> confidence that our changes will not introduce critical issues (as long as
> the system is covered by the test). It also forces us to be more rigorous in
> our design and implementation. The hope for the Ozone community is that we
> can have distributed system testing as a crucial baseline skill for all Ozone
> community members. If in the future, we decide to introduce a new component
> or rework Ozone (e.g. porting it to Rust, etc), I hope that it's built on the
> distributed system testing foundation (e.g. we build systems on top of DST
> first) to ensure that Ozone correctness remain preserved. It's going to be
> difficult, but I believe it's worth it long term.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
