[
https://issues.apache.org/jira/browse/HDDS-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai reassigned HDDS-8727:
--------------------------------------
Assignee: Attila Doroszlai
> EC Reconstruction creates multiple commands for each target DN in new RM
> ------------------------------------------------------------------------
>
> Key: HDDS-8727
> URL: https://issues.apache.org/jira/browse/HDDS-8727
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Varsha Ravi
> Assignee: Attila Doroszlai
> Priority: Major
>
> With the new replication manager and throttling feature, if there are 2
> missing replicas in a EC container, the expectation is to create a single EC
> reconstruction command and send it to a target DN which will act as
> coordinator. But there are multiple reconstruction commands being sent to
> each individual target DN.
>
> {noformat}
> 2023-05-30 07:14:00,537 [EventQueue-DeadNodeForDeadNodeHandler] INFO
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected.
> bd768c8b-1f30-41a5-b989-40fa5ba07da7
> 2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of
> size 19 for DN bd768c8b-1f30-41a5-b989-40fa5ba07da7
> 2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node:
> /default/bd768c8b-1f30-41a5-b989-40fa5ba07da7
> 2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected.
> 116c39e4-4e96-4158-b609-3498d1175abe
> 2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of
> size 21 for DN 116c39e4-4e96-4158-b609-3498d1175abe
> 2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node:
> /default/116c39e4-4e96-4158-b609-3498d1175abe
> 2023-05-30 07:17:30,597 [Under Replicated Processor] INFO
> org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending
> command [reconstructECContainersCommand: containerID: 54, replicationConfig:
> EC{rs-3-2-1024k}, sources: [d50d3938-3054-4471-a0bd-5d48ea0b7f95
> replicaIndex: 2, 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4,
> 59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets:
> [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988], missingIndexes: [1]] for container
> ContainerInfo{id=#54, state=CLOSED,
> pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
> stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to
> e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 with datanode deadline 1685432820597 and
> scm deadline 1685432850597
> 2023-05-30 07:17:30,597 [Under Replicated Processor] WARN
> org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler:
> Exception while creating the replication or reconstruction commands for
> container #54.
> 2023-05-30 07:17:30,597 [Under Replicated Processor] ERROR
> org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor:
> Error processing Health result of class: class
> org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
> for container ContainerInfo{id=#54, state=CLOSED,
> pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
> stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
> 2023-05-30 07:21:00,646 [Under Replicated Processor] WARN
> org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler:
> Exception while creating the replication or reconstruction commands for
> container #54.
> 2023-05-30 07:21:00,646 [Under Replicated Processor] ERROR
> org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor:
> Error processing Health result of class: class
> org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
> for container ContainerInfo{id=#54, state=CLOSED,
> pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
> stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
> 2023-05-30 07:21:30,655 [Under Replicated Processor] INFO
> org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending
> command [reconstructECContainersCommand: containerID: 54, replicationConfig:
> EC{rs-3-2-1024k}, sources: [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988
> replicaIndex: 1, d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex: 2,
> 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4,
> 59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets:
> [4ee90fec-7046-4c07-bd1e-8f6e3ae20520], missingIndexes: [3]] for container
> ContainerInfo{id=#54, state=CLOSED,
> pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99,
> stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to
> 4ee90fec-7046-4c07-bd1e-8f6e3ae20520 with datanode deadline 1685433060655 and
> scm deadline 1685433090655{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
