Wei-Chiu Chuang created HDDS-15802:
--------------------------------------
Summary: Add integration/system tests to certify per-volume push
replication thread pools on DataNode (HDDS-15412).
Key: HDDS-15802
URL: https://issues.apache.org/jira/browse/HDDS-15802
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Wei-Chiu Chuang
h2. Background
[HDDS-15412|https://github.com/apache/ozone/pull/10693] adds optional
per-volume push replication thread pools on the DataNode:
* {{hdds.datanode.replication.per.volume.enabled}} (default: {{{}false{}}})
* {{hdds.datanode.replication.per.volume.streams.limit}} (default: {{{}1{}}},
reconfigurable)
When enabled, push replication is routed to a volume-specific executor so slow
replication on one disk does not block replication on other disks. Pull
replication, EC reconstruction, and other tasks continue using the global
replication pool. Failed volumes shut down their per-volume pool via
{{{}shutdownFailedVolumePools(){}}}.
Unit tests in {{TestReplicationSupervisor}} cover init, routing, isolation,
resize, and volume-failure shutdown. Cluster-level certification is still
missing.
h2. Motivation
Operators enable this feature primarily during decommission/maintenance on
multi-volume datanodes. We need integration tests that prove:
# Basic replication still works with the feature enabled.
# The feature delivers its intended isolation benefit under realistic cluster
load.
# The datanode recovers correctly from volume failure and SCM failover while
replication is in flight.
Without these tests, we rely on unit coverage and manual validation before
enabling the feature in production.
h2. Scope
Add integration tests under {{{}hadoop-ozone/integration-test{}}}, preferably
in a new class:
{{org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication}}
Extend existing suites where appropriate:
* {{TestDatanodeReconfiguration}} — per-volume pool resize via reconfig
* Optionally {{TestDatanodeHddsVolumeFailureDetection}} / volume-failure
patterns
h3. In scope
||Priority||Test||Description||
|P0|Push/pull regression with per-volume enabled|Multi-volume DN
({{{}setNumDataVolumes(2){}}}), feature ON. Verify existing push/pull
replication from {{TestContainerReplication}} still succeeds. Assert volume
pools are created (count matches DN volume count).|
|P0|Volume failure: healthy-volume replication continues|Fail one volume on a
multi-volume DN while per-volume pools are enabled. Assert failed-volume pool
is removed; push replication for containers on healthy volumes still succeeds;
DN remains within toleration limits.|
|P0|Decommission E2E with per-volume pools|Multi-volume DN with Ratis and/or EC
data; feature ON with {{{}streams.limit=1{}}}. Decommission the DN. Assert node
reaches {{{}DECOMMISSIONED{}}}, under-replicated containers recover, and
cluster remains healthy.|
|P1|Runtime reconfiguration of per-volume pool size|Extend
{{{}TestDatanodeReconfiguration{}}}: assert
{{hdds.datanode.replication.per.volume.streams.limit}} is reconfigurable;
reconfigure (e.g. 1 → 3) and verify all volume pool sizes update.|
|P1|Cross-volume push isolation under concurrent load|Containers on volume A
and B on same source DN; {{{}streams.limit=1{}}}. Slow push on A should not
block push completion on B.|
|P1|SCM failover during active replication|SCM HA cluster (3 SCMs), per-volume
enabled, under-replication in progress (decommission or DN stop).
{{transferLeadership()}} mid-replication. Assert containers eventually recover,
DN {{getTotalInFlightReplications()}} returns to 0, and volume pools remain
functional.|
|P2|Default-off backward compatibility|Same regression tests with
{{{}per.volume.enabled=false{}}}; assert {{getVolumeReplicationThreadPools()}}
is null and behavior matches pre-feature baseline.|
|P2|Pull replication uses global pool|With feature ON, pull replication
succeeds; optional log/assertion that pull tasks are not routed to per-volume
executors.|
|P2|Decommission/maintenance pool scaling|DN enters {{{}DECOMMISSIONING{}}};
assert per-volume pool size scales per {{outofservice.limit.factor}} (mirrors
unit test {{{}perVolumePoolResizeOnNodeStateChange{}}}).|
h3. Out of scope
* Re-testing unit-level behavior already covered in
{{TestReplicationSupervisor}} (init logging, config validation, non-push
routing).
* SCM-side reconstruction counter behavior on failover (HDDS-15071).
* Toggling {{per.volume.enabled}} at runtime without restart (not
reconfigurable).
* Acceptance/compose-cluster tests (can be a follow-up if mini-cluster
coverage is insufficient).
* Performance/benchmark tests.
h2. Acceptance criteria
* New integration test class (or clearly scoped extensions to existing
classes) covers all P0 scenarios.
* At least one P1 scenario implemented (recommend: SCM failover or
cross-volume isolation).
* All new tests pass in CI with {{{}-DskipShade -DskipRecon -DskipDocs{}}}.
* Tests use short SCM/DN report intervals consistent with existing
replication integration tests.
* No new third-party dependencies.
* Checkstyle and RAT pass.
h2. Implementation notes
* Use {{MiniOzoneCluster.newBuilder(conf).setNumDataVolumes(2)}} (see
{{{}TestDatanodeHddsVolumeFailureToleration{}}}).
* For SCM failover, use {{MiniOzoneHAClusterImpl}} and
{{transferLeadership()}} (see {{{}TestFailoverWithSCMHA{}}},
{{{}TestDatanodeSCMNodesReconfiguration{}}}).
* Access supervisor via
{{{}datanode.getDatanodeStateMachine().getSupervisor(){}}}.
* Pool assertions via {{supervisor.getVolumeReplicationThreadPools()}} and
{{{}getPoolSize(volumeRoot){}}}.
* Reuse helpers from {{TestContainerReplication}} for direct push/pull command
injection where appropriate.
* For decommission E2E, follow patterns from
{{TestDecommissionAndMaintenance}} / {{{}TestReplicationManagerIntegration{}}}.
* Consider schema V2/V3 parameterization if other DN integration tests do
({{{}@ContainerLayoutTestInfo{}}}).
h2. Minimum viable delivery
If timeboxed, land P0 only:
# Push/pull regression with feature ON
# Volume failure continuity
# Decommission E2E
Add SCM failover as the first P1 follow-up.
h2. References
* Implementation PR: [#10693|https://github.com/apache/ozone/pull/10693]
* Parent/feature JIRA: HDDS-15412
* Related design context: EC decommission reconstruction (per-volume DN pools
are Phase 3 enabler)
* Existing integration patterns:
** {{TestContainerReplication}}
** {{TestDatanodeHddsVolumeFailureDetection}}
** {{TestDecommissionAndMaintenance}}
** {{TestFailoverWithSCMHA}}
** {{TestDatanodeReconfiguration}}
h2. Test plan (for PR description)
mvn -pl :ozone-integration-test test \
-Dtest=TestPerVolumePushReplication\
-DskipShade-DskipRecon-DskipDocs
Also run extended {{TestDatanodeReconfiguration}} if modified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
