yandrey321 commented on PR #11727: URL: https://github.com/apache/ozone/pull/11727#issuecomment-6429655876
> When volume failed, why it is not triggering FCR? According to https://issues.apache.org/jira/browse/HDDS-8448, it should trigger FCR right. Upon processing FCR, aren't we closing the open containers for replication? It does trigger the FCR: `ContainerSet.handleVolumeFailures` still sends it right away, as HDDS-8448 added (`getFullContainerReportDiscardPendingICR()` → `refreshFullReport` → `triggerHeartbeat`), and this PR doesn't change that. The problem is that processing the FCR doesn't close an OPEN EC container: 1. **The FCR only removes the replica.** The lost container is missing from the report, so `ContainerReportHandler.processMissingReplicas` removes that datanode's replica from SCM (`removeContainerReplica`) and from the node's container set, and stops there. There is no close and no replication command. 2. **The replication manager doesn't treat the container as unhealthy.** For an OPEN container, `OpenContainerHandler` closes it only if it has no healthy pipeline, or if one of the *remaining* replicas isn't OPEN (`isOpenContainerHealthy` checks `allMatch` over the replicas that are left). After the FCR, the 4 remaining replicas of the RS(3,2) container are all OPEN. The EC pipeline is still healthy too, because its datanodes are alive and heartbeating, so the container counts as healthy and stays OPEN. 3. **An OPEN container isn't replicated anyway.** `OpenContainerHandler.handle` returns `true` for any OPEN container, which stops the handler chain, so under-replication isn't checked until the container is CLOSED. That is why it has to be closed first. For Ratis the close doesn't come from the FCR either. The next write to the lost container fails in the Ratis state machine, and the datanode asks SCM to close the pipeline. EC writes go over gRPC and have no such path, so SCM kept allocating the container, and every write to the missing index failed with `CONTAINER_MISSING`. The integration test shows this. With the volume-failure hook removed, `TestECContainerCloseOnVolumeFailure` still sends the FCR, yet times out with the container still OPEN in SCM. With the hook, the CLOSE action goes out in the same heartbeat as the FCR. The container then moves to CLOSING and CLOSED, and the missing index is reconstructed. An alternative would be for SCM to close an OPEN EC container whose replica count falls below the required nodes. But SCM can't tell a lost replica from a newly allocated container whose replicas haven't been reported yet, so it could close new containers by mistake. Only the datanode knows that it lost the replica, so it reports it, the same way the Ratis path does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
