Elavarasan Kathirvel created HDDS-17580:
-------------------------------------------
Summary: SCM container info fails 5+ min after restart with
"Failed to lookup datanode" from NodeManager
Key: HDDS-17580
URL: https://issues.apache.org/jira/browse/HDDS-17580
Project: Apache Ozone
Issue Type: Bug
Reporter: Elavarasan Kathirvel
After a full SCM HA restart, if a container’s pipeline still references a
datanode that is not in SCM NodeManager (for example, a datanode stopped before
restart), {{ozone admin container info <id> --json}} fails instead of returning
metadata. {{{}Failed to lookup datanode in NodeManager{}}}, and Replication
Manager stays idle for a long interval after safemode exit. so stale
replica/pipeline state is reconciled slowly and container info remains
unavailable for several minutes.
h3. Steps to reproduce
# Create a key.
# Run the key information command to retrieve the container ID.
# Run the container information command to retrieve the list of replica
DataNode hosts.
# Stop one of the replica DataNodes.
# Restart the SCM service on all SCM hosts.
# Wait until SCM exits safemode.
# Run the Ozone admin container information command again.
# Verify whether the command returns the expected {{Failed to lookup
datanode}} error.
h3. Expected behavior
* Within ~1 minute after SCM leaves safemode, {{ozone admin container info
<id> --json}} should succeed and return valid JSON, even when a replica
datanode is stopped or not yet present in NodeManager.
h3. Actual behavior
* For 5+ minutes after SCM restart and safemode exit, {{container info}} fails
with {{Failed to lookup datanode in NodeManager.}}
{code:java}
[root@elavarasan-1 ~]# ozone sh volume create test
[root@elavarasan-1 ~]# ozone sh bucket create test/bucket
[root@elavarasan-1 ~]#
[root@elavarasan-1 ~]# ls
anaconda-ks.cfg anaconda-post.log anaconda-post-nochroot.log buildinfo
original-ks.cfg
[root@elavarasan-1 ~]# ozone sh key put test/bucket/key /root/anaconda-ks.cfg
[root@elavarasan-1 ~]# ozone sh key info test/bucket/key
{
"volumeName" : "test",
"bucketName" : "bucket",
"name" : "key",
"owner" : "hdfs",
"dataSize" : 8977,
"creationTime" : "2026-10-23T13:44:30.939Z",
"modificationTime" : "2026-10-23T13:44:31.945Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"minimumNodes" : 1,
"replicationType" : "RATIS"
},
"metadata" : { },
"tags" : { },
"ozoneKeyLocations" : [ {
"containerID" : 3,
"localID" : 117889302528001035,
"length" : 8977,
"offset" : 0,
"keyOffset" : 0
} ],
"generation" : 291,
"file" : true
}
[root@elavarasan-1 ~]#
[root@elavarasan-1 ~]#
[root@elavarasan-1 ~]# ozone admin container info 3
RemoteException: SCM Server, while invoking StorageContainerLocationProtocolPB
over node1. Retrying in 2000ms after 0 failover attempt(s).
Container id: 3
Pipeline id: e0177ca1-2218-468a-9faa-335c49389358
Write PipelineId: e0177ca1-2218-468a-9faa-335c49389358
Write Pipeline State: OPEN
Container State: OPEN
SequenceId: 139
Datanodes: [fc72f929-d579-4501-92d7-e47d15c34125/elavarasan-9,
e66b1f4d-f328-4c5f-9526-1fde7011ea42/elavarasan-8,
332752c0-c6eb-4273-911e-559671ad6dc8/elavarasan-3]
Replicas: [State: OPEN; ReplicaIndex: 0; SequenceId: 139; Origin:
332752c0-c6eb-4273-911e-559671ad6dc8; Location:
332752c0-c6eb-4273-911e-559671ad6dc8/elavarasan-3,
State: OPEN; ReplicaIndex: 0; SequenceId: 139; Origin:
e66b1f4d-f328-4c5f-9526-1fde7011ea42; Location:
e66b1f4d-f328-4c5f-9526-1fde7011ea42/elavarasan-8,
State: OPEN; ReplicaIndex: 0; SequenceId: 139; Origin:
fc72f929-d579-4501-92d7-e47d15c34125; Location:
fc72f929-d579-4501-92d7-e47d15c34125/elavarasan-9]
[root@elavarasan-1 ~]# ozone admin container info 3
Failed to lookup datanode 332752c0-c6eb-4273-911e-559671ad6dc8(elavarasan-3) in
NodeManager{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
