[
https://issues.apache.org/jira/browse/HDDS-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17022:
------------------------------
Attachment: TestBugMC6RecoveringReservationLeak.java
> RECOVERING container (EC reconstruction target) keeps its space reservation
> after it is closed, deleted or marked unhealthy
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17022
> URL: https://issues.apache.org/jira/browse/HDDS-17022
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-6-release-reservation-when-leaving-recovering.patch,
> TestBugMC6RecoveringReservationLeak.java
>
>
> h3. Mechanism
> Since HDDS-12810 every CreateContainer reserves the container size in
> {{VolumeChoosingPolicy.chooseVolume}}, and {{KeyValueContainer.create}} marks
> the container as holding a reservation, whatever its state. A RECOVERING
> container is writable and its writes consume the reservation through
> {{ContainerData.incrWriteBytes}}, so holding one is expected. But
> {{ContainerData.setState}} gives the reservation back only when the old state
> is OPEN. A RECOVERING container therefore keeps the unwritten part of its
> reservation when it moves to CLOSING and CLOSED, when it is force deleted
> ({{markContainerForDelete}} sets DELETED) and when the stale recovering
> scrubber marks it UNHEALTHY. Nothing recomputes {{HddsVolume.committedBytes}}
> while the datanode runs. This is a regression from HDDS-12810 (2.1.0): before
> it, only OPEN containers reserved (in {{ContainerSet.addContainer}} through
> {{commitSpace}}), so RECOVERING containers held nothing to release.
> h3. Trigger
> Normal EC reconstruction, no fault: {{ECReconstructionCoordinator}} creates
> RECOVERING containers on the targets and then closes them, or deletes them
> with force after a failed reconstruction.
> h3. Impact
> Each reconstruction leaves the unwritten part of a container size committed
> on the target volume until restart; a failed one, deleted early, leaves
> nearly all of it. On a datanode that receives many reconstructions (for
> example after the loss of a datanode or disk holding many EC replicas, or
> repeated reconstruction failures), volumes end up refusing new containers and
> imports while empty, and the inflated committed value is reported in the
> storage report.
> h3. Reproduction
> [^TestBugMC6RecoveringReservationLeak.java]
> ({{hadoop-hdds/container-service}}) uses a real {{HddsVolume}} with a mocked
> 1000 MB usage source, 300 MB containers, the real
> {{RoundRobinVolumeChoosingPolicy}} and {{KeyValueHandler}}, and sends the
> CreateContainer (state RECOVERING), CloseContainer and forced DeleteContainer
> requests the EC coordinator sends. A closed RECOVERING container keeps
> {{committed=314572800}}, also after it is deleted, while an OPEN control
> container returns to 0. After three create and delete cycles the empty volume
> has {{committed=943718400}} and a new OPEN container gets
> {{DISK_OUT_OF_SPACE}}. A passing test means the defect is present. Not
> exercised: a real reconstruction on a cluster, and the SCM side.
> h3. Patch
> [^MC-6-release-reservation-when-leaving-recovering.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3. Released 2.1.0 and later are affected.
> {{ContainerData.setState}} releases when the old state is OPEN or RECOVERING
> ({{HddsUtils.isOpenToWriteState}}), and {{ContainerData.commitSpace}} accepts
> the same states, so both sides agree on which containers hold a reservation.
> The release is guarded by the reservation flag, so containers that never
> reserved (imports, containers loaded at restart, DiskBalancer temporary data)
> are not affected, and one change covers close, delete and the scrubber. The
> {{commitSpace}} change has no effect at load ({{ContainerReader}} returns
> before it for RECOVERING containers); it lets a RECOVERING container reserve
> again, for example when a failed state change is rolled back. Covered by the
> new {{TestKeyValueHandler.testRecoveringContainerReleasesReservation}} for
> the three exit paths, which also checks that a RECOVERING container can
> reserve again. It fails without the change, and also with only the
> {{commitSpace}} half (5368709120 still committed after the exit). With it the
> handler, container, reader, scrubber, DiskBalancer, persistence, close
> command and importer suites pass (760 tests) and checkstyle is clean.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
