[
https://issues.apache.org/jira/browse/HDDS-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-16627:
--------------------------------
Description:
SCM bugs that involve datanode failures, decommission or maintenance, and SCM
restarts are hard to find and hard to reproduce. Whether they happen depends on
the order in which the node health check, the event handlers,
ReplicationManager and the pipeline services see each change. MiniOzoneCluster
tests are too slow to try many such orders, and a failed run usually cannot be
repeated in the same order.
This adds a deterministic simulation test for SCM, TestScmSimulation in
hdds-server-scm. It runs the real StorageContainerManager in-process against
simulated datanodes:
* Simulated datanodes register and heartbeat through SCM's datanode protocol
server, send container and pipeline reports, and carry out the commands SCM
sends back.
* SCM is built with a simulated clock, seeded random sources, and an event
queue whose handlers run on the simulation's scheduler. The simulation runs
SCM's background services itself: the node health check, ReplicationManager,
the background pipeline creator, the pipeline scrubber and the decommission
monitor.
* Everything runs on one thread, so a run is fully determined by its seed.
Rerunning a seed gives exactly the same trace. Three hours of simulated time
take about 1-2 seconds.
* During the chaos phase, a client writes to containers while datanodes crash,
get partitioned, are decommissioned (one at a time or a whole rack), enter
maintenance or join, replicas are lost or corrupted, and SCM restarts. Then the
faults stop, every datanode comes back, and SCM gets time to repair the cluster.
* Checked during the run: SCM never deletes the last replica, the last healthy
replica, or a replica newer than every other healthy one; open containers are
on open pipelines; healthy datanodes are in the network topology; no event
handler or service throws.
* Checked at the end: every container is correctly replicated, every
decommission and maintenance finishes, SCM and the datanodes agree on the
replicas, and SCM sends no replicate or delete commands for the next 15
simulated minutes.
Not simulated: SCM HA, block deletion, EC containers, the container balancer
and security.
A few production classes change so that the test can inject what it needs:
* SCMConfigurator can set the event queue, the clock, the pipeline ID
generator, and the Random that ReplicationManager uses to shuffle vulnerable
UNHEALTHY replicas.
* The clock is passed down to the node manager and to the container and
pipeline state maps.
* NetworkTopologyImpl can take a seeded Random, and PipelinePlacementPolicy is
no longer final.
The default run uses seeds 1, 2 and 3. To rerun a failed seed, or to try new
ones:
{noformat}
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
-Dscm.simulation.seed=SEED
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
-Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random
{noformat}
Bugs found with this test are linked to this issue as "Discovered while
testing".
was:
SCM bugs that involve datanode failures, decommission or maintenance, and SCM
restarts are hard to find and hard to reproduce. Whether they happen depends on
the order in which the node health check, the event handlers,
ReplicationManager and the pipeline services see each change. MiniOzoneCluster
tests are too slow to try many such orders, and a failed run usually cannot be
repeated in the same order.
This adds a deterministic simulation test for SCM, TestScmSimulation in
hdds-server-scm. It runs the real StorageContainerManager in-process against
simulated datanodes:
* Simulated datanodes register and heartbeat through SCM's datanode protocol
server, send container and pipeline reports, and carry out the commands SCM
sends back.
* SCM is built with a simulated clock, seeded random sources, and an event
queue whose handlers run on the simulation's scheduler. The simulation runs
SCM's background services itself: the node health check, ReplicationManager,
the background pipeline creator, the pipeline scrubber and the decommission
monitor.
* Everything runs on one thread, so a run is fully determined by its seed.
Rerunning a seed gives exactly the same trace. Three hours of simulated time
take about 1-2 seconds.
* During the chaos phase, a client writes to containers while datanodes crash,
get partitioned, are decommissioned (one at a time or a whole rack), enter
maintenance or join, replicas are lost or corrupted, and SCM restarts. Then the
faults stop, every datanode comes back, and SCM gets time to repair the cluster.
* Checked during the run: SCM never deletes the last replica, the last healthy
replica, or a replica newer than every other healthy one; open containers are
on open pipelines; healthy datanodes are in the network topology; no event
handler or service throws.
* Checked at the end: every container is correctly replicated, every
decommission and maintenance finishes, SCM and the datanodes agree on the
replicas, and SCM sends no replicate or delete commands for the next 15
simulated minutes.
Not simulated: SCM HA, block deletion, EC containers, the container balancer
and security.
A few production classes change so that the test can inject what it needs:
* SCMConfigurator can set the event queue, the clock and the pipeline ID
generator.
* The clock is passed down to the node manager and to the container and
pipeline state maps.
* NetworkTopologyImpl can take a seeded Random, and PipelinePlacementPolicy is
no longer final.
* RatisUnderReplicationHandler rotates the vulnerable UNHEALTHY replicas
instead of shuffling them.
The default run uses seeds 1, 2 and 3. To rerun a failed seed, or to try new
ones:
{noformat}
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
-Dscm.simulation.seed=SEED
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
-Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random
{noformat}
Bugs found with this test are linked to this issue as "Discovered while
testing".
> Add a deterministic simulation test for SCM
> -------------------------------------------
>
> Key: HDDS-16627
> URL: https://issues.apache.org/jira/browse/HDDS-16627
> Project: Apache Ozone
> Issue Type: Test
> Components: SCM
> Reporter: Chu Cheng Li
> Priority: Major
>
> SCM bugs that involve datanode failures, decommission or maintenance, and SCM
> restarts are hard to find and hard to reproduce. Whether they happen depends
> on the order in which the node health check, the event handlers,
> ReplicationManager and the pipeline services see each change.
> MiniOzoneCluster tests are too slow to try many such orders, and a failed run
> usually cannot be repeated in the same order.
> This adds a deterministic simulation test for SCM, TestScmSimulation in
> hdds-server-scm. It runs the real StorageContainerManager in-process against
> simulated datanodes:
> * Simulated datanodes register and heartbeat through SCM's datanode protocol
> server, send container and pipeline reports, and carry out the commands SCM
> sends back.
> * SCM is built with a simulated clock, seeded random sources, and an event
> queue whose handlers run on the simulation's scheduler. The simulation runs
> SCM's background services itself: the node health check, ReplicationManager,
> the background pipeline creator, the pipeline scrubber and the decommission
> monitor.
> * Everything runs on one thread, so a run is fully determined by its seed.
> Rerunning a seed gives exactly the same trace. Three hours of simulated time
> take about 1-2 seconds.
> * During the chaos phase, a client writes to containers while datanodes
> crash, get partitioned, are decommissioned (one at a time or a whole rack),
> enter maintenance or join, replicas are lost or corrupted, and SCM restarts.
> Then the faults stop, every datanode comes back, and SCM gets time to repair
> the cluster.
> * Checked during the run: SCM never deletes the last replica, the last
> healthy replica, or a replica newer than every other healthy one; open
> containers are on open pipelines; healthy datanodes are in the network
> topology; no event handler or service throws.
> * Checked at the end: every container is correctly replicated, every
> decommission and maintenance finishes, SCM and the datanodes agree on the
> replicas, and SCM sends no replicate or delete commands for the next 15
> simulated minutes.
> Not simulated: SCM HA, block deletion, EC containers, the container balancer
> and security.
> A few production classes change so that the test can inject what it needs:
> * SCMConfigurator can set the event queue, the clock, the pipeline ID
> generator, and the Random that ReplicationManager uses to shuffle vulnerable
> UNHEALTHY replicas.
> * The clock is passed down to the node manager and to the container and
> pipeline state maps.
> * NetworkTopologyImpl can take a seeded Random, and PipelinePlacementPolicy
> is no longer final.
> The default run uses seeds 1, 2 and 3. To rerun a failed seed, or to try new
> ones:
> {noformat}
> mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
> -Dscm.simulation.seed=SEED
> mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos
> -Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random
> {noformat}
> Bugs found with this test are linked to this issue as "Discovered while
> testing".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
