[
https://issues.apache.org/jira/browse/HDDS-15802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095407#comment-18095407
]
Wei-Chiu Chuang commented on HDDS-15802:
----------------------------------------
Update – this PR will focus on P0 items.
P1 items goes to HDDS-15815
P2 items goes to HDDS-15816
> Add integration/system tests to certify per-volume push replication thread
> pools on DataNode (HDDS-15412).
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15802
> URL: https://issues.apache.org/jira/browse/HDDS-15802
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Labels: pull-request-available
>
> h2. Background
> [HDDS-15412|https://github.com/apache/ozone/pull/10693] adds optional
> per-volume push replication thread pools on the DataNode:
> * {{hdds.datanode.replication.per.volume.enabled}} (default: {{{}false{}}})
> * {{hdds.datanode.replication.per.volume.streams.limit}} (default:
> {{{}1{}}}, reconfigurable)
> When enabled, push replication is routed to a volume-specific executor so
> slow replication on one disk does not block replication on other disks. Pull
> replication, EC reconstruction, and other tasks continue using the global
> replication pool. Failed volumes shut down their per-volume pool via
> {{{}shutdownFailedVolumePools(){}}}.
> Unit tests in {{TestReplicationSupervisor}} cover init, routing, isolation,
> resize, and volume-failure shutdown. Cluster-level certification is still
> missing.
> h2. Motivation
> Operators enable this feature primarily during decommission/maintenance on
> multi-volume datanodes. We need integration tests that prove:
> # Basic replication still works with the feature enabled.
> # The feature delivers its intended isolation benefit under realistic
> cluster load.
> # The datanode recovers correctly from volume failure and SCM failover while
> replication is in flight.
> Without these tests, we rely on unit coverage and manual validation before
> enabling the feature in production.
> h2. Scope
> Add integration tests under {{{}hadoop-ozone/integration-test{}}}, preferably
> in a new class:
> {{org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication}}
> Extend existing suites where appropriate:
> * {{TestDatanodeReconfiguration}} — per-volume pool resize via reconfig
> * Optionally {{TestDatanodeHddsVolumeFailureDetection}} / volume-failure
> patterns
> h3. In scope
> ||Priority||Test||Description||
> |P0|Push/pull regression with per-volume enabled|Multi-volume DN
> ({{{}setNumDataVolumes(2){}}}), feature ON. Verify existing push/pull
> replication from {{TestContainerReplication}} still succeeds. Assert volume
> pools are created (count matches DN volume count).|
> |P0|Volume failure: healthy-volume replication continues|Fail one volume on a
> multi-volume DN while per-volume pools are enabled. Assert failed-volume pool
> is removed; push replication for containers on healthy volumes still
> succeeds; DN remains within toleration limits.|
> |P0|Decommission E2E with per-volume pools|Multi-volume DN with Ratis and/or
> EC data; feature ON with {{{}streams.limit=1{}}}. Decommission the DN. Assert
> node reaches {{{}DECOMMISSIONED{}}}, under-replicated containers recover, and
> cluster remains healthy.|
> |P1|Runtime reconfiguration of per-volume pool size|Extend
> {{{}TestDatanodeReconfiguration{}}}: assert
> {{hdds.datanode.replication.per.volume.streams.limit}} is reconfigurable;
> reconfigure (e.g. 1 → 3) and verify all volume pool sizes update.|
> |P1|Cross-volume push isolation under concurrent load|Containers on volume A
> and B on same source DN; {{{}streams.limit=1{}}}. Slow push on A should not
> block push completion on B.|
> |P1|SCM failover during active replication|SCM HA cluster (3 SCMs),
> per-volume enabled, under-replication in progress (decommission or DN stop).
> {{transferLeadership()}} mid-replication. Assert containers eventually
> recover, DN {{getTotalInFlightReplications()}} returns to 0, and volume pools
> remain functional.|
> |P2|Default-off backward compatibility|Same regression tests with
> {{{}per.volume.enabled=false{}}}; assert
> {{getVolumeReplicationThreadPools()}} is null and behavior matches
> pre-feature baseline.|
> |P2|Pull replication uses global pool|With feature ON, pull replication
> succeeds; optional log/assertion that pull tasks are not routed to per-volume
> executors.|
> |P2|Decommission/maintenance pool scaling|DN enters {{{}DECOMMISSIONING{}}};
> assert per-volume pool size scales per {{outofservice.limit.factor}} (mirrors
> unit test {{{}perVolumePoolResizeOnNodeStateChange{}}}).|
> h3. Out of scope
> * Re-testing unit-level behavior already covered in
> {{TestReplicationSupervisor}} (init logging, config validation, non-push
> routing).
> * SCM-side reconstruction counter behavior on failover (HDDS-15071).
> * Toggling {{per.volume.enabled}} at runtime without restart (not
> reconfigurable).
> * Acceptance/compose-cluster tests (can be a follow-up if mini-cluster
> coverage is insufficient).
> * Performance/benchmark tests.
> h2. Acceptance criteria
> * New integration test class (or clearly scoped extensions to existing
> classes) covers all P0 scenarios.
> * At least one P1 scenario implemented (recommend: SCM failover or
> cross-volume isolation).
> * All new tests pass in CI with {{{}-DskipShade -DskipRecon -DskipDocs{}}}.
> * Tests use short SCM/DN report intervals consistent with existing
> replication integration tests.
> * No new third-party dependencies.
> * Checkstyle and RAT pass.
> h2. Implementation notes
> * Use {{MiniOzoneCluster.newBuilder(conf).setNumDataVolumes(2)}} (see
> {{{}TestDatanodeHddsVolumeFailureToleration{}}}).
> * For SCM failover, use {{MiniOzoneHAClusterImpl}} and
> {{transferLeadership()}} (see {{{}TestFailoverWithSCMHA{}}},
> {{{}TestDatanodeSCMNodesReconfiguration{}}}).
> * Access supervisor via
> {{{}datanode.getDatanodeStateMachine().getSupervisor(){}}}.
> * Pool assertions via {{supervisor.getVolumeReplicationThreadPools()}} and
> {{{}getPoolSize(volumeRoot){}}}.
> * Reuse helpers from {{TestContainerReplication}} for direct push/pull
> command injection where appropriate.
> * For decommission E2E, follow patterns from
> {{TestDecommissionAndMaintenance}} /
> {{{}TestReplicationManagerIntegration{}}}.
> * Consider schema V2/V3 parameterization if other DN integration tests do
> ({{{}@ContainerLayoutTestInfo{}}}).
> h2. Minimum viable delivery
> If timeboxed, land P0 only:
> # Push/pull regression with feature ON
> # Volume failure continuity
> # Decommission E2E
> Add SCM failover as the first P1 follow-up.
> h2. References
> * Implementation PR: [#10693|https://github.com/apache/ozone/pull/10693]
> * Parent/feature JIRA: HDDS-15412
> * Related design context: EC decommission reconstruction (per-volume DN
> pools are Phase 3 enabler)
> * Existing integration patterns:
> ** {{TestContainerReplication}}
> ** {{TestDatanodeHddsVolumeFailureDetection}}
> ** {{TestDecommissionAndMaintenance}}
> ** {{TestFailoverWithSCMHA}}
> ** {{TestDatanodeReconfiguration}}
> h2. Test plan (for PR description)
> mvn -pl :ozone-integration-test test \
> -Dtest=TestPerVolumePushReplication\
> -DskipShade-DskipRecon-DskipDocs
> Also run extended {{TestDatanodeReconfiguration}} if modified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
