Siyao Meng created HDDS-17009:
---------------------------------
Summary: Container create retried on another volume after an
IOException leaves the full container size reserved on the abandoned volume
Key: HDDS-17009
URL: https://issues.apache.org/jira/browse/HDDS-17009
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-release-reservation-on-create-retry.patch,
TestBugMC5CreateRetryReservationLeak.java
h3. Mechanism
{{VolumeChoosingPolicy.chooseVolume}} reserves the container size on the volume
it returns ({{RoundRobinVolumeChoosingPolicy}},
{{CapacityVolumeChoosingPolicy}}, since HDDS-12810). When
{{KeyValueContainer.create}} then gets an {{IOException}} from
{{createContainerMetaData}} on that volume, it deletes the container directory,
removes the volume from the candidate list and tries the next volume, without
releasing the reservation. The next attempt overwrites
{{containerData.volume}}, so the release in
{{KeyValueHandler.handleCreateContainer}} (only on failure, through
{{releaseCommitSpace}}) can only reach the last volume tried. Nothing
recomputes {{HddsVolume.committedBytes}} while the datanode runs.
h3. Trigger
A datanode with two or more volumes. A CreateContainer whose first chosen
volume fails to create the container directories or DB with an {{IOException}}
(EIO, ENOSPC, EACCES), and the retry on another volume succeeds. If every
volume fails in the directory or DB creation, all but the last one leak.
h3. Impact
Each failed attempt keeps the full container size (5 GB by default) committed
on a volume that holds no such container, until the datanode restarts. After
the fault clears, the volume keeps refusing new containers and imports it has
room for ({{AvailableSpaceFilter}} subtracts the committed bytes), and the
inflated committed value is reported in the storage report. No data is lost.
h3. Reproduction
[^TestBugMC5CreateRetryReservationLeak.java]
({{hadoop-hdds/container-service}}) uses two real volumes with a fixed 1000 MB
capacity, 600 MB containers, the real {{RoundRobinVolumeChoosingPolicy}} and
{{KeyValueHandler}}. The fault is a real filesystem error: the first volume's
cluster directory is read only. Container 1 ends on the second volume and the
first keeps {{committed=629145600}}. After the fault is cleared and container 1
is closed, two creates that fit one per volume give {{SUCCESS}} and
{{DISK_OUT_OF_SPACE}}, with the first volume empty. A passing test means the
defect is present. Not exercised: a real EIO or ENOSPC, and the SCM side.
h3. Patch
[^MC-5-release-reservation-on-create-retry.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3. Released 2.1.0 and later have the same code.
The retry branch calls {{containerData.releaseCommitSpace()}} before it moves
to the next volume. At that point the container data still points at the failed
volume and nothing is written, so exactly the reserved size is released; the
flag it clears makes the handler's later release a no op, so nothing is
released twice. Covered by the existing
{{TestKeyValueContainer.testNextVolumeTriedOnWriteFailure}}, whose mocked
policy now reserves like the real ones and which now checks the committed bytes
of both volumes, and by the new
{{testReservationReleasedWhenCreateFailsOnAllVolumes}}. Both fail without the
change (5368709120 committed on a failed volume). With it the container and
handler suites pass (485 tests) and checkstyle is clean.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
