[
https://issues.apache.org/jira/browse/HDDS-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722791#comment-17722791
]
Attila Doroszlai edited comment on HDDS-8603 at 5/15/23 2:06 PM:
-----------------------------------------------------------------
Looks like this may happen even without concurrency. Failed previously in
[TestContainerReplication|https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/15/20796/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.replication.TestContainerReplication-output.txt],
which replicates containers serially.
was (Author: adoroszlai):
Looks like this may happen even without concurrency. Failed previously in
https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/15/20796/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.replication.TestContainerReplication-output.txt
> Push replication may fail with StatusRuntimeException: UNAVAILABLE: Channel
> shutdown invoked
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-8603
> URL: https://issues.apache.org/jira/browse/HDDS-8603
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Affects Versions: 1.4.0
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Major
> Attachments:
> org.apache.hadoop.ozone.scm.node.TestDecommissionAndMaintenance-output.txt,
> org.apache.hadoop.ozone.scm.node.TestDecommissionAndMaintenance.txt
>
>
> {code:title=SCM}
> 2023-05-12 06:37:25,691 [Under Replicated Processor] INFO
> replication.ReplicationManager
> (ReplicationManager.java:sendDatanodeCommand(684)) - Sending command
> [replicateContainerCommand: containerId=11, replicaIndex=1,
> targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47),
> priority=NORMAL] for container ContainerInfo{id=#11, state=CLOSED,
> pipelineID=PipelineID=1fdad600-ca58-422a-8cbc-f3f0ea92deb5,
> stateEnterTime=2023-05-12T06:37:20.327Z, owner=om1} to
> 1aeadef1-6c90-4225-addb-3772c43da32e(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47)
> with datanode deadline 1683875215691 and scm deadline 1683875245691
> 2023-05-12 06:37:25,691 [Under Replicated Processor] INFO
> replication.ReplicationManager
> (ReplicationManager.java:sendDatanodeCommand(684)) - Sending command
> [replicateContainerCommand: containerId=10, replicaIndex=4,
> targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47),
> priority=NORMAL] for container ContainerInfo{id=#10, state=CLOSED,
> pipelineID=PipelineID=f8d093fe-a02d-4415-b965-bb0272b84054,
> stateEnterTime=2023-05-12T06:37:19.680Z, owner=om1} to
> 1aeadef1-6c90-4225-addb-3772c43da32e(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47)
> with datanode deadline 1683875215691 and scm deadline 1683875245691
> {code}
> {code:title=source datanode}
> 2023-05-12 06:37:27,364 [ContainerReplicationThread-1] INFO
> replication.PushReplicator (PushReplicator.java:replicate(58)) - Starting
> replication of container 11 to
> dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47)
> using NO_COMPRESSION
> 2023-05-12 06:37:27,367 [ContainerReplicationThread-2] INFO
> replication.PushReplicator (PushReplicator.java:replicate(58)) - Starting
> replication of container 10 to
> dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47)
> using NO_COMPRESSION
> 2023-05-12 06:37:27,384 [ContainerReplicationThread-1] INFO
> replication.GrpcOutputStream (GrpcOutputStream.java:close(111)) - Sent 13824
> bytes for container 11
> 2023-05-12 06:37:27,407 [grpc-default-executor-7] WARN
> replication.GrpcContainerUploader (GrpcContainerUploader.java:onError(113)) -
> Failed to upload container 11 to
> dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47)
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE:
> Channel shutdown invoked
> ...
> 2023-05-12 06:37:27,407 [ContainerReplicationThread-1] WARN
> replication.PushReplicator (PushReplicator.java:replicate(73)) - Container 11
> replication was unsuccessful.
> 2023-05-12 06:37:27,408 [ContainerReplicationThread-1] WARN
> replication.ReplicationSupervisor (ReplicationSupervisor.java:run(350)) -
> Failed FAILED replicateContainerCommand: containerId=11, replicaIndex=1,
> targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47),
> priority=NORMAL, transferred 13824 bytes
> ...
> 2023-05-12 06:37:27,411 [ContainerReplicationThread-2] INFO
> replication.GrpcOutputStream (GrpcOutputStream.java:close(111)) - Sent 12800
> bytes for container 10
> 2023-05-12 06:37:27,439 [ContainerReplicationThread-2] INFO
> replication.ReplicationSupervisor (ReplicationSupervisor.java:run(353)) -
> Successful DONE replicateContainerCommand: containerId=10, replicaIndex=4,
> targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47),
> priority=NORMAL, transferred 12800 bytes
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
