[ 
https://issues.apache.org/jira/browse/HDDS-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766352#comment-17766352
 ] 

Stephen O'Donnell commented on HDDS-6236:
-----------------------------------------

There has been some disussion about this, and a suggesting that if an 
unexpected container is reported to SCM, we should just add it to the SCM 
RocksDB as if it is supposed to exist. 

Adding the container to SCM is not as simple as it seems.

First, if the reported container has a containerID ahead of the current SCM 
containerID, then it cannot be simply added, as when the SCM containerID 
sequence is advanced to the same as the reported container there will be a 
conflict.

Second, the reported container replica does not contain details of its 
ReplicationConfig. If does contain the replicaIndex, so we can determine if it 
is EC or not EC, as EC will always have a replica index greater than 0. However 
without the repConfig, we don't know the EC scheme, so its unreadable.

We can also guess a Ratis container is replication=THREE, but it might be ONE 
etc.

HDFS just deletes reported blocks which have no reference in the namespace.

Ozone has a config to log a warning (the default) or behave like HDFS.

My feeling is that when this happens, its likely due to bad data left behind on 
a datanode from an earlier install. However we could perhaps deal with these 
unexpected containers in Recon more sensibly. Recon could receive the container 
and highlight if its an unexpected container. If it is, check if the 
containerID is referenced anywhere in the OM namespace. If its not, just ignore 
it. If it is, present a warning about an unexpected container reported which is 
also referenced in OM.

However, even that is not perfect - what if the current containerID is 10, and 
we get an unexpected container reported with an ID of 20. At this stage, if you 
checked OM, the container will not be expected. However then the system 
advances to create container ID 20. At this stage, we could have 3 replicas 
with ID 20, and then this other old replica also with ID 20. Now we some way to 
deal with the bad replica and how to identify it from the others. I am not sure 
what would happen here - probably RM would notice the container has 4 replicas 
and one is closed, so it will close the pipeline. Then the "bad" container 
could have a later BCSID, and the system may take it as the "best" replica, 
remove the others and replicate it.

So, there appears to be many edge cases here - my preference would be to act 
like HDFS - if we don't expect it, get rid of it by deleting it before it 
causes other problems or potential for dataloss later!


> SCM receives reports of unknown containers 
> -------------------------------------------
>
>                 Key: HDDS-6236
>                 URL: https://issues.apache.org/jira/browse/HDDS-6236
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: SCM
>            Reporter: Ethan Rose
>            Priority: Major
>
> We have noticed the following log messages in SCM leader and followers for 
> multiple containers:
> {code:java}
> 2022-01-19 12:53:24,021 ERROR 
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Received 
> container report for an unknown container 1368 from datanode \{ ... }
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: ID #1368
>         at 
> org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.lambda$getContainer$0(ContainerManagerImpl.java:147)
>         at java.base/java.util.Optional.orElseThrow(Optional.java:408)
>         at 
> org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.getContainer(ContainerManagerImpl.java:147)
>         at 
> org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:94)
>         at 
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:165)
>         at 
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:133)
>         at 
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:48)
>         at 
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
>         at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>         at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>         at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> The cluster is currently running SCM HA, but the issue was observed when it 
> was a non-HA cluster as well. This seems to only affect empty containers, 
> since no data appears to be missing. Containers are supposed to exist in SCM 
> DB even after they have been deleted from the datanode, so there seems to be 
> some kind of bug in the container persistence logic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to