Chu Cheng Li created HDDS-16627:
-----------------------------------
Summary: Add a deterministic simulation test for SCM
Key: HDDS-16627
URL: https://issues.apache.org/jira/browse/HDDS-16627
Project: Apache Ozone
Issue Type: Test
Components: SCM
Reporter: Chu Cheng Li
SCM bugs that involve datanode failures, decommission or maintenance, and SCM
restarts are hard to find and hard to reproduce. Whether they happen depends on
the order in which the node health check, the event handlers,
ReplicationManager and the pipeline services see each change. MiniOzoneCluster
tests are too slow to try many such orders, and a failed run usually cannot be
repeated in the same order.
This adds a deterministic simulation test for SCM, TestScmSimulation in
hdds-server-scm. It runs the real StorageContainerManager in-process against
simulated datanodes:
* Simulated datanodes register and heartbeat through SCM's datanode protocol
server, send container and pipeline reports, and carry out the commands SCM
sends back.
* SCM is built with a simulated clock, seeded random sources, and an event
queue whose handlers run on the simulation's scheduler. The simulation runs
SCM's background services itself: the node health check, ReplicationManager,
the background pipeline creator, the pipeline scrubber and the decommission
monitor.
* Everything runs on one thread, so a run is fully determined by its seed.
Rerunning a seed gives exactly the same trace. Three hours of simulated time
take about 1-2 seconds.
* During the chaos phase, a client writes to containers while datanodes crash,
get partitioned, are decommissioned (one at a time or a whole rack), enter
maintenance or join, replicas are lost or corrupted, and SCM restarts. Then the
faults stop, every datanode comes back, and SCM gets time to repair the cluster.
* Checked during the run: SCM never deletes the last replica, the last healthy
replica, or a replica newer than every other healthy one; open containers are
on open pipelines; healthy datanodes are in the network topology; no event
handler or service throws.
* Checked at the end: every container is correctly replicated, every
decommission and maintenance finishes, SCM and the datanodes agree on the
replicas, and SCM sends no replicate or delete commands for the next 15
simulated minutes.
Not simulated: SCM HA, block deletion, EC containers, the container balancer
and security.
A few production classes change so that the test can inject what it needs:
* SCMConfigurator can set the event queue, the clock and the pipeline ID
generator.
* The clock is passed down to the node manager and to the container and
pipeline state maps.
* NetworkTopologyImpl can take a seeded Random, and PipelinePlacementPolicy is
no longer final.
* RatisUnderReplicationHandler rotates the vulnerable UNHEALTHY replicas
instead of shuffling them.
The default run uses seeds 1, 2 and 3. To rerun a failed seed, or to try new
ones:
{noformat}
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
-Dscm.simulation.seed=SEED
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
-Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random
{noformat}
Bugs found with this test are linked to this issue as "Discovered while
testing".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
