[
https://issues.apache.org/jira/browse/HDDS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17009:
------------------------------
Attachment: TestBugMC5CreateRetryReservationLeak.java
> Container create retried on another volume after an IOException leaves the
> full container size reserved on the abandoned volume
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17009
> URL: https://issues.apache.org/jira/browse/HDDS-17009
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-5-release-reservation-on-create-retry.patch,
> TestBugMC5CreateRetryReservationLeak.java
>
>
> h3. Mechanism
> {{VolumeChoosingPolicy.chooseVolume}} reserves the container size on the
> volume it returns ({{RoundRobinVolumeChoosingPolicy}},
> {{CapacityVolumeChoosingPolicy}}, since HDDS-12810). When
> {{KeyValueContainer.create}} then gets an {{IOException}} from
> {{createContainerMetaData}} on that volume, it deletes the container
> directory, removes the volume from the candidate list and tries the next
> volume, without releasing the reservation. The next attempt overwrites
> {{containerData.volume}}, so the release in
> {{KeyValueHandler.handleCreateContainer}} (only on failure, through
> {{releaseCommitSpace}}) can only reach the last volume tried. Nothing
> recomputes {{HddsVolume.committedBytes}} while the datanode runs.
> h3. Trigger
> A datanode with two or more volumes. A CreateContainer whose first chosen
> volume fails to create the container directories or DB with an
> {{IOException}} (EIO, ENOSPC, EACCES), and the retry on another volume
> succeeds. If every volume fails in the directory or DB creation, all but the
> last one leak.
> h3. Impact
> Each failed attempt keeps the full container size (5 GB by default) committed
> on a volume that holds no such container, until the datanode restarts. After
> the fault clears, the volume keeps refusing new containers and imports it has
> room for ({{AvailableSpaceFilter}} subtracts the committed bytes), and the
> inflated committed value is reported in the storage report. No data is lost.
> h3. Reproduction
> [^TestBugMC5CreateRetryReservationLeak.java]
> ({{hadoop-hdds/container-service}}) uses two real volumes with a fixed 1000
> MB capacity, 600 MB containers, the real {{RoundRobinVolumeChoosingPolicy}}
> and {{KeyValueHandler}}. The fault is a real filesystem error: the first
> volume's cluster directory is read only. Container 1 ends on the second
> volume and the first keeps {{committed=629145600}}. After the fault is
> cleared and container 1 is closed, two creates that fit one per volume give
> {{SUCCESS}} and {{DISK_OUT_OF_SPACE}}, with the first volume empty. A passing
> test means the defect is present. Not exercised: a real EIO or ENOSPC, and
> the SCM side.
> h3. Patch
> [^MC-5-release-reservation-on-create-retry.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3. Released 2.1.0 and later have the same code.
> The retry branch calls {{containerData.releaseCommitSpace()}} before it moves
> to the next volume. At that point the container data still points at the
> failed volume and nothing is written, so exactly the reserved size is
> released; the flag it clears makes the handler's later release a no op, so
> nothing is released twice. Covered by the existing
> {{TestKeyValueContainer.testNextVolumeTriedOnWriteFailure}}, whose mocked
> policy now reserves like the real ones and which now checks the committed
> bytes of both volumes, and by the new
> {{testReservationReleasedWhenCreateFailsOnAllVolumes}}. Both fail without the
> change (5368709120 committed on a failed volume). With it the container and
> handler suites pass (485 tests) and checkstyle is clean.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
