[
https://issues.apache.org/jira/browse/HDDS-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17580:
----------------------------------
Labels: pull-request-available (was: )
> SCM container info fails 5+ min after restart with "Failed to lookup
> datanode" from NodeManager
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-17580
> URL: https://issues.apache.org/jira/browse/HDDS-17580
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Elavarasan Kathirvel
> Assignee: Sarveksha Yeshavantha Raju
> Priority: Critical
> Labels: pull-request-available
>
> After a full SCM HA restart, if a container’s pipeline still references a
> datanode that is not in SCM NodeManager (for example, a datanode stopped
> before restart), {{ozone admin container info <id> --json}} fails instead of
> returning metadata. {{{}Failed to lookup datanode in NodeManager{}}}, and
> Replication Manager stays idle for a long interval after safemode exit. so
> stale replica/pipeline state is reconciled slowly and container info remains
> unavailable for several minutes.
> h3. Steps to reproduce
> # Create a key.
> # Run the key information command to retrieve the container ID.
> # Run the container information command to retrieve the list of replica
> DataNode hosts.
> # Stop one of the replica DataNodes.
> # Restart the SCM service on all SCM hosts.
> # Wait until SCM exits safemode.
> # Run the Ozone admin container information command again.
> # Verify whether the command returns the expected {{Failed to lookup
> datanode}} error.
> h3. Expected behavior
> * Within ~1 minute after SCM leaves safemode, {{ozone admin container info
> <id> --json}} should succeed and return valid JSON, even when a replica
> datanode is stopped or not yet present in NodeManager.
> h3. Actual behavior
> * For 5+ minutes after SCM restart and safemode exit, {{container info}}
> fails with {{Failed to lookup datanode in NodeManager.}}
>
>
> {code:java}
> [root@elavarasan-1 ~]# ozone sh volume create test
> [root@elavarasan-1 ~]# ozone sh bucket create test/bucket
> [root@elavarasan-1 ~]#
> [root@elavarasan-1 ~]# ls
> anaconda-ks.cfg anaconda-post.log anaconda-post-nochroot.log
> buildinfo original-ks.cfg
> [root@elavarasan-1 ~]# ozone sh key put test/bucket/key /root/anaconda-ks.cfg
> [root@elavarasan-1 ~]# ozone sh key info test/bucket/key
> {
> "volumeName" : "test",
> "bucketName" : "bucket",
> "name" : "key",
> "owner" : "hdfs",
> "dataSize" : 8977,
> "creationTime" : "2026-10-23T13:44:30.939Z",
> "modificationTime" : "2026-10-23T13:44:31.945Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "ozoneKeyLocations" : [ {
> "containerID" : 3,
> "localID" : 117889302528001035,
> "length" : 8977,
> "offset" : 0,
> "keyOffset" : 0
> } ],
> "generation" : 291,
> "file" : true
> }
> [root@elavarasan-1 ~]#
> [root@elavarasan-1 ~]#
> [root@elavarasan-1 ~]# ozone admin container info 3
> RemoteException: SCM Server, while invoking
> StorageContainerLocationProtocolPB over node1. Retrying in 2000ms after 0
> failover attempt(s).
> Container id: 3
> Pipeline id: e0177ca1-2218-468a-9faa-335c49389358
> Write PipelineId: e0177ca1-2218-468a-9faa-335c49389358
> Write Pipeline State: OPEN
> Container State: OPEN
> SequenceId: 139
> Datanodes: [fc72f929-d579-4501-92d7-e47d15c34125/elavarasan-9,
> e66b1f4d-f328-4c5f-9526-1fde7011ea42/elavarasan-8,
> 332752c0-c6eb-4273-911e-559671ad6dc8/elavarasan-3]
> Replicas: [State: OPEN; ReplicaIndex: 0; SequenceId: 139; Origin:
> 332752c0-c6eb-4273-911e-559671ad6dc8; Location:
> 332752c0-c6eb-4273-911e-559671ad6dc8/elavarasan-3,
> State: OPEN; ReplicaIndex: 0; SequenceId: 139; Origin:
> e66b1f4d-f328-4c5f-9526-1fde7011ea42; Location:
> e66b1f4d-f328-4c5f-9526-1fde7011ea42/elavarasan-8,
> State: OPEN; ReplicaIndex: 0; SequenceId: 139; Origin:
> fc72f929-d579-4501-92d7-e47d15c34125; Location:
> fc72f929-d579-4501-92d7-e47d15c34125/elavarasan-9]
> [root@elavarasan-1 ~]# ozone admin container info 3
> Failed to lookup datanode 332752c0-c6eb-4273-911e-559671ad6dc8(elavarasan-3)
> in NodeManager{code}
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
