yandrey321 opened a new pull request, #11727:
URL: https://github.com/apache/ozone/pull/11727
## What changes were proposed in this pull request?
**Problem.** When a datanode loses the replica of an open EC container, for
example because its volume failed, nothing closes the container, and SCM keeps
allocating it:
- the datanode moves the container to its missing set and sends a full
container report, and writes to it fail with `CONTAINER_MISSING` ("has been
lost and cannot be recreated");
- SCM drops the replica, but `OpenContainerHandler` only checks that each
*remaining* replica is OPEN, so an open EC container with a missing index
counts as healthy;
- the container never fills, because the writes fail, so it isn't closed for
being full either.
Ratis containers on the same volume are closed within seconds: the failed
write makes the Ratis state machine unhealthy, and the datanode asks SCM to
close the pipeline. EC writes go through the gRPC path, which has no such
handling.
**Implementation idea.** The datanode asks SCM to close the container,
through the existing `ContainerAction` CLOSE
in the heartbeat, in two places:
- **When a volume fails** (`ContainerSet.handleVolumeFailures`), for each
OPEN EC container on it (replica index > 0).
The existing code already triggers a heartbeat after the full container
report, so SCM closes the container
within seconds, whether or not anyone writes to it.
- **When a write over gRPC to a missing container fails with
`CONTAINER_MISSING`** (`HddsDispatcher`), if it is an EC
write (replica index > 0 on WriteChunk or PutBlock). Only the first such
write per container triggers an immediate
heartbeat; later ones re-add the action, deduplicated while pending.
SCM's existing path does the rest:
- `ContainerActionsHandler` fires `CLOSE_CONTAINER`;
- the container moves to CLOSING and is detached from its pipeline, so it's
no longer allocated, and the EC pipeline
is closed at the next allocation;
- the replicas are force-closed;
- once CLOSED, the missing index is reconstructed.
**Key decisions.**
- **Datanode-driven, like Ratis.** Only the datanode that lost the replica
reports it, so a newly allocated container
whose replicas haven't been created yet is never closed by mistake. That
risk would come with an SCM-side check on
replica counts.
handles every CLOSE
action the same way, and the datanode log says why.
- **Two triggers.** The volume-failure hook closes containers that nobody
writes to. The write hook covers an action
lost with a failed heartbeat, SCM failover, and containers that went
missing in other ways.
**Side effects.**
- **More, harmless CLOSE actions.** Each failed EC write to a lost container
re-adds the action. SCM ignores CLOSE
for containers that are no longer OPEN, so repeats and reports from
several datanodes are harmless.
- **Earlier close.** Open EC containers on a failed volume are now closed
even if no one writes to them, so their
missing index gets reconstructed sooner.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17360
## How was this patch tested?
Integration and unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
