Siyao Meng created HDDS-17022:
---------------------------------
Summary: RECOVERING container (EC reconstruction target) keeps its
space reservation after it is closed, deleted or marked unhealthy
Key: HDDS-17022
URL: https://issues.apache.org/jira/browse/HDDS-17022
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
Since HDDS-12810 every CreateContainer reserves the container size in
{{VolumeChoosingPolicy.chooseVolume}}, and {{KeyValueContainer.create}} marks
the container as holding a reservation, whatever its state. A RECOVERING
container is writable and its writes consume the reservation through
{{ContainerData.incrWriteBytes}}, so holding one is expected. But
{{ContainerData.setState}} gives the reservation back only when the old state
is OPEN. A RECOVERING container therefore keeps the unwritten part of its
reservation when it moves to CLOSING and CLOSED, when it is force deleted
({{markContainerForDelete}} sets DELETED) and when the stale recovering
scrubber marks it UNHEALTHY. Nothing recomputes {{HddsVolume.committedBytes}}
while the datanode runs. This is a regression from HDDS-12810 (2.1.0): before
it, only OPEN containers reserved (in {{ContainerSet.addContainer}} through
{{commitSpace}}), so RECOVERING containers held nothing to release.
h3. Trigger
Normal EC reconstruction, no fault: {{ECReconstructionCoordinator}} creates
RECOVERING containers on the targets and then closes them, or deletes them with
force after a failed reconstruction.
h3. Impact
Each reconstruction leaves the unwritten part of a container size committed on
the target volume until restart; a failed one, deleted early, leaves nearly all
of it. On a datanode that receives many reconstructions (for example after the
loss of a datanode or disk holding many EC replicas, or repeated reconstruction
failures), volumes end up refusing new containers and imports while empty, and
the inflated committed value is reported in the storage report.
h3. Reproduction
[^TestBugMC6RecoveringReservationLeak.java] ({{hadoop-hdds/container-service}})
uses a real {{HddsVolume}} with a mocked 1000 MB usage source, 300 MB
containers, the real {{RoundRobinVolumeChoosingPolicy}} and
{{KeyValueHandler}}, and sends the CreateContainer (state RECOVERING),
CloseContainer and forced DeleteContainer requests the EC coordinator sends. A
closed RECOVERING container keeps {{committed=314572800}}, also after it is
deleted, while an OPEN control container returns to 0. After three create and
delete cycles the empty volume has {{committed=943718400}} and a new OPEN
container gets {{DISK_OUT_OF_SPACE}}. A passing test means the defect is
present. Not exercised: a real reconstruction on a cluster, and the SCM side.
h3. Patch
[^MC-6-release-reservation-when-leaving-recovering.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3. Released 2.1.0 and later are affected.
{{ContainerData.setState}} releases when the old state is OPEN or RECOVERING
({{HddsUtils.isOpenToWriteState}}), and {{ContainerData.commitSpace}} accepts
the same states, so both sides agree on which containers hold a reservation.
The release is guarded by the reservation flag, so containers that never
reserved (imports, containers loaded at restart, DiskBalancer temporary data)
are not affected, and one change covers close, delete and the scrubber. The
{{commitSpace}} change has no effect at load ({{ContainerReader}} returns
before it for RECOVERING containers); it lets a RECOVERING container reserve
again, for example when a failed state change is rolled back. Covered by the
new {{TestKeyValueHandler.testRecoveringContainerReleasesReservation}} for the
three exit paths, which also checks that a RECOVERING container can reserve
again. It fails without the change, and also with only the {{commitSpace}} half
(5368709120 still committed after the exit). With it the handler, container,
reader, scrubber, DiskBalancer, persistence, close command and importer suites
pass (760 tests) and checkstyle is clean.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
