[
https://issues.apache.org/jira/browse/HDDS-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16627:
----------------------------------
Labels: pull-request-available (was: )
> Add a deterministic simulation test for SCM
> -------------------------------------------
>
> Key: HDDS-16627
> URL: https://issues.apache.org/jira/browse/HDDS-16627
> Project: Apache Ozone
> Issue Type: Test
> Components: SCM
> Reporter: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> SCM bugs that involve datanode failures, decommission or maintenance, and SCM
> restarts are hard to find and hard to reproduce. Whether they happen depends
> on the order in which the node health check, the event handlers,
> ReplicationManager and the pipeline services see each change.
> MiniOzoneCluster tests are too slow to try many such orders, and a failed run
> usually cannot be repeated in the same order.
> This adds a deterministic simulation test for SCM, TestScmSimulation in
> hdds-server-scm. It runs the real StorageContainerManager in-process against
> simulated datanodes:
> * Simulated datanodes register and heartbeat through SCM's datanode protocol
> server, send container and pipeline reports, and carry out the commands SCM
> sends back.
> * SCM is built with a simulated clock, seeded random sources, and an event
> queue whose handlers run on the simulation's scheduler. The simulation runs
> SCM's background services itself: the node health check, ReplicationManager,
> the background pipeline creator, the pipeline scrubber and the decommission
> monitor.
> * Everything runs on one thread, so a run is fully determined by its seed.
> Rerunning a seed gives exactly the same trace. Three hours of simulated time
> take about 1-2 seconds.
> * During the chaos phase, a client writes to containers while datanodes
> crash, get partitioned, are decommissioned (one at a time or a whole rack),
> enter maintenance or join, replicas are lost or corrupted, and SCM restarts.
> Then the faults stop, every datanode comes back, and SCM gets time to repair
> the cluster.
> * Checked during the run: SCM never deletes the last replica, the last
> healthy replica, or a replica newer than every other healthy one; open
> containers are on open pipelines; healthy datanodes are in the network
> topology; no event handler or service throws.
> * Checked at the end: every container is correctly replicated, every
> decommission and maintenance finishes, SCM and the datanodes agree on the
> replicas, and SCM sends no replicate or delete commands for the next 15
> simulated minutes.
> Not simulated: SCM HA, block deletion, EC containers, the container balancer
> and security.
> A few production classes change so that the test can inject what it needs:
> * SCMConfigurator can set the event queue, the clock, the pipeline ID
> generator, and the Random that ReplicationManager uses to shuffle vulnerable
> UNHEALTHY replicas.
> * The clock is passed down to the node manager and to the container and
> pipeline state maps.
> * NetworkTopologyImpl can take a seeded Random, and PipelinePlacementPolicy
> is no longer final.
> The default run uses seeds 1, 2 and 3. To rerun a failed seed, or to try new
> ones:
> {noformat}
> mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
> -Dscm.simulation.seed=SEED
> mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
> -Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random
> {noformat}
> Bugs found with this test are linked to this issue as "Discovered while
> testing".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
