[
https://issues.apache.org/jira/browse/HDDS-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gargi Jaiswal reassigned HDDS-17024:
------------------------------------
Assignee: Gargi Jaiswal
> Failed .container write during mark for close rolls the container back to
> OPEN without its space reservation
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17024
> URL: https://issues.apache.org/jira/browse/HDDS-17024
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Gargi Jaiswal
> Priority: Minor
> Attachments: MC-8-keep-reservation-on-failed-mark-for-close.patch,
> TestBugMC8MarkForCloseRollbackReservation.java
>
>
> h3. Mechanism
> {{KeyValueContainer.updateContainerData}} applies the state change first and
> writes the {{.container}} file second. For mark for close,
> {{ContainerData.setState(CLOSING)}} leaves OPEN and releases the container's
> reservation ({{maxSize - bytesUsed}}) and its reservation flag. When the file
> write then fails, the rollback {{setState(oldState)}} restores OPEN but does
> not reserve again. The container stays OPEN and writable with nothing
> reserved; its writes are not charged against the volume either. This is a
> regression from HDDS-12810 (2.1.0), which removed the reservation
> {{setState}} used to make on every transition back to OPEN.
> h3. Trigger
> SCM sends CloseContainer (or the pipeline is removed) and writing the
> container's {{.container}} file fails with an {{IOException}} (EIO, ENOSPC,
> EACCES) while the volume stays healthy. {{CloseContainerCommandHandler}} only
> logs the error, so the container remains OPEN until the close is retried
> successfully.
> h3. Impact
> While the container stays OPEN, the volume's committed bytes lack its
> headroom, so {{AvailableSpaceFilter}} admits new containers and imports the
> volume has no room for, and the storage report understates the committed
> space. A later successful close or a restart corrects the counter, but
> containers admitted in the meantime stay.
> h3. Reproduction
> [^TestBugMC8MarkForCloseRollbackReservation.java]
> ({{hadoop-hdds/container-service}}) uses a real {{HddsVolume}} with a mocked
> 1000 MB usage source, 600 MB containers, the real
> {{RoundRobinVolumeChoosingPolicy}} and {{KeyValueHandler}}. The fault is a
> real filesystem error: the container's metadata directory is read only (run
> as a non root user). A second create is refused with {{DISK_OUT_OF_SPACE}}
> before the fault. After the failed mark for close: {{state=OPEN
> committedSpace=false committed=0}}, and the same second create succeeds,
> leaving two OPEN containers with 1200 MB of headroom on a 1000 MB volume. A
> passing test means the defect is present. Not exercised: a real EIO or ENOSPC.
> h3. Patch
> [^MC-8-keep-reservation-on-failed-mark-for-close.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3. Released 2.1.0 and later are affected.
> {{updateContainerData}} remembers whether space was reserved before the
> update; if the rollback leaves it unreserved, it calls
> {{ContainerData.commitSpace()}}, which reserves the current {{maxSize -
> bytesUsed}} again. The fix is in the rollback because that is the only place
> a container legitimately returns to OPEN, and the check of the earlier flag
> avoids reserving for an OPEN container that held nothing before (which the
> removed unconditional {{commitSpace}} in {{setState}} did not check). Covered
> by the new
> {{TestKeyValueContainer.testReservationKeptWhenMarkForCloseFails}}, which
> also checks that a retried mark for close then releases the space; it fails
> without the change (no reservation after the rollback). With it the
> container, handler, reader, persistence, close command, state machine and
> stale recovering scrubber suites pass (564 tests) and checkstyle is clean.
> RECOVERING containers get the same rollback once HDDS-17022 makes them
> release on leaving RECOVERING and lets {{commitSpace}} reserve for them.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
