Varsha Ravi created HDDS-8727:
---------------------------------
Summary: EC Reconstruction creates multiple commands for each
target DN in new RM
Key: HDDS-8727
URL: https://issues.apache.org/jira/browse/HDDS-8727
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Varsha Ravi
With the new replication manager and throttling feature, if there are 2 missing
replicas in a EC container, the expectation is to create a single EC
reconstruction command and send it to a target DN which will act as
coordinator. But there are multiple reconstruction commands being sent to each
individual target DN.
{noformat}
2023-05-30 07:14:00,537 [EventQueue-DeadNodeForDeadNodeHandler] INFO
org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected.
bd768c8b-1f30-41a5-b989-40fa5ba07da7
2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO
org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of size
19 for DN bd768c8b-1f30-41a5-b989-40fa5ba07da7
2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node:
/default/bd768c8b-1f30-41a5-b989-40fa5ba07da7
2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO
org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected.
116c39e4-4e96-4158-b609-3498d1175abe
2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO
org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of size
21 for DN 116c39e4-4e96-4158-b609-3498d1175abe
2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node:
/default/116c39e4-4e96-4158-b609-3498d1175abe
2023-05-30 07:17:30,597 [Under Replicated Processor] INFO
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending
command [reconstructECContainersCommand: containerID: 54, replicationConfig:
EC{rs-3-2-1024k}, sources: [d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex:
2, 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4,
59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets:
[e29c8a8f-9164-4bdd-b2ef-82c9e6b22988], missingIndexes: [1]] for container
ContainerInfo{id=#54, state=CLOSED,
pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to
e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 with datanode deadline 1685432820597 and
scm deadline 1685432850597
2023-05-30 07:17:30,597 [Under Replicated Processor] WARN
org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler:
Exception while creating the replication or reconstruction commands for
container #54.
2023-05-30 07:17:30,597 [Under Replicated Processor] ERROR
org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor:
Error processing Health result of class: class
org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
for container ContainerInfo{id=#54, state=CLOSED,
pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
2023-05-30 07:21:00,646 [Under Replicated Processor] WARN
org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler:
Exception while creating the replication or reconstruction commands for
container #54.
2023-05-30 07:21:00,646 [Under Replicated Processor] ERROR
org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor:
Error processing Health result of class: class
org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
for container ContainerInfo{id=#54, state=CLOSED,
pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
2023-05-30 07:21:30,655 [Under Replicated Processor] INFO
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending
command [reconstructECContainersCommand: containerID: 54, replicationConfig:
EC{rs-3-2-1024k}, sources: [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 replicaIndex:
1, d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex: 2,
86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4,
59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets:
[4ee90fec-7046-4c07-bd1e-8f6e3ae20520], missingIndexes: [3]] for container
ContainerInfo{id=#54, state=CLOSED,
pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to
4ee90fec-7046-4c07-bd1e-8f6e3ae20520 with datanode deadline 1685433060655 and
scm deadline 1685433090655{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
