[
https://issues.apache.org/jira/browse/HDDS-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16780:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in
> TestPerVolumePushReplication#testDecommissionWithPerVolumePools
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-16780
> URL: https://issues.apache.org/jira/browse/HDDS-16780
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 82.46
> s <<< FAILURE! -- in
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
> [ERROR]
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools
> -- Time elapsed: 31.25 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools(TestPerVolumePushReplication.java:271)
> {code}
> SCM log during the decommission (197 occurrences):
> {code}
> ERROR replication.UnhealthyReplicationProcessor
> (UnhealthyReplicationProcessor.java:processAll(125)) - Error processing
> Health result of class: class
> org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
> ...
> java.lang.IllegalArgumentException: Capacity cannot be negative.
> at
> org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:45)
> at
> org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1172)
> at
> org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:1147)
> at
> org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.excludeFullNodes(ReplicationManagerUtil.java:245)
> at
> org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.getExcludedAndUsedNodes(ReplicationManagerUtil.java:223)
> at
> org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndSendCommands(ECUnderReplicationHandler.java:130)
> {code}
> h3. Root cause
> The integration test {{ozone-site.xml}} sets
> {{hdds.datanode.du.factory.classname}} to
> {{MockSpaceUsageCheckFactory$None}}, which reports {{Long.MAX_VALUE}}
> capacity for every volume. {{TestPerVolumePushReplication}} starts datanodes
> with 2 data volumes, so {{SCMNodeManager#getNodeStatInternal}} overflows when
> it sums the storage reports of a node, and {{SCMNodeStat}} throws {{Capacity
> cannot be negative}}.
> {{ReplicationManagerUtil#excludeFullNodes}} calls {{getNodeStat}} for every
> datanode with a pending replica ADD. So while any replication is in flight,
> processing of every other under replicated container fails, and the
> decommission effectively replicates one container per ReplicationManager
> cycle (about 3s). The decommissioning node held 12 containers; it had reached
> 9 of them when the 30s {{waitForDnToReachOpState}} wait expired.
> h3. Fix
> Use {{MockSpaceUsageCheckFactory$HalfTera}} (fixed 512 GB per volume) in this
> test, so the per node sum stays in range.
> - https://github.com/apache/ozone/actions/runs/37686266652/job/113019002301
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
