[
https://issues.apache.org/jira/browse/HDDS-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766352#comment-17766352
]
Stephen O'Donnell commented on HDDS-6236:
-----------------------------------------
There has been some disussion about this, and a suggesting that if an
unexpected container is reported to SCM, we should just add it to the SCM
RocksDB as if it is supposed to exist.
Adding the container to SCM is not as simple as it seems.
First, if the reported container has a containerID ahead of the current SCM
containerID, then it cannot be simply added, as when the SCM containerID
sequence is advanced to the same as the reported container there will be a
conflict.
Second, the reported container replica does not contain details of its
ReplicationConfig. If does contain the replicaIndex, so we can determine if it
is EC or not EC, as EC will always have a replica index greater than 0. However
without the repConfig, we don't know the EC scheme, so its unreadable.
We can also guess a Ratis container is replication=THREE, but it might be ONE
etc.
HDFS just deletes reported blocks which have no reference in the namespace.
Ozone has a config to log a warning (the default) or behave like HDFS.
My feeling is that when this happens, its likely due to bad data left behind on
a datanode from an earlier install. However we could perhaps deal with these
unexpected containers in Recon more sensibly. Recon could receive the container
and highlight if its an unexpected container. If it is, check if the
containerID is referenced anywhere in the OM namespace. If its not, just ignore
it. If it is, present a warning about an unexpected container reported which is
also referenced in OM.
However, even that is not perfect - what if the current containerID is 10, and
we get an unexpected container reported with an ID of 20. At this stage, if you
checked OM, the container will not be expected. However then the system
advances to create container ID 20. At this stage, we could have 3 replicas
with ID 20, and then this other old replica also with ID 20. Now we some way to
deal with the bad replica and how to identify it from the others. I am not sure
what would happen here - probably RM would notice the container has 4 replicas
and one is closed, so it will close the pipeline. Then the "bad" container
could have a later BCSID, and the system may take it as the "best" replica,
remove the others and replicate it.
So, there appears to be many edge cases here - my preference would be to act
like HDFS - if we don't expect it, get rid of it by deleting it before it
causes other problems or potential for dataloss later!
> SCM receives reports of unknown containers
> -------------------------------------------
>
> Key: HDDS-6236
> URL: https://issues.apache.org/jira/browse/HDDS-6236
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Ethan Rose
> Priority: Major
>
> We have noticed the following log messages in SCM leader and followers for
> multiple containers:
> {code:java}
> 2022-01-19 12:53:24,021 ERROR
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Received
> container report for an unknown container 1368 from datanode \{ ... }
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: ID #1368
> at
> org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.lambda$getContainer$0(ContainerManagerImpl.java:147)
> at java.base/java.util.Optional.orElseThrow(Optional.java:408)
> at
> org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.getContainer(ContainerManagerImpl.java:147)
> at
> org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:94)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:165)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:133)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:48)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> The cluster is currently running SCM HA, but the issue was observed when it
> was a non-HA cluster as well. This seems to only affect empty containers,
> since no data appears to be missing. Containers are supposed to exist in SCM
> DB even after they have been deleted from the datanode, so there seems to be
> some kind of bug in the container persistence logic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]