Siyao Meng created HDDS-17024:
---------------------------------
Summary: Failed .container write during mark for close rolls the
container back to OPEN without its space reservation
Key: HDDS-17024
URL: https://issues.apache.org/jira/browse/HDDS-17024
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-8-keep-reservation-on-failed-mark-for-close.patch,
TestBugMC8MarkForCloseRollbackReservation.java
h3. Mechanism
{{KeyValueContainer.updateContainerData}} applies the state change first and
writes the {{.container}} file second. For mark for close,
{{ContainerData.setState(CLOSING)}} leaves OPEN and releases the container's
reservation ({{maxSize - bytesUsed}}) and its reservation flag. When the file
write then fails, the rollback {{setState(oldState)}} restores OPEN but does
not reserve again. The container stays OPEN and writable with nothing reserved;
its writes are not charged against the volume either. This is a regression from
HDDS-12810 (2.1.0), which removed the reservation {{setState}} used to make on
every transition back to OPEN.
h3. Trigger
SCM sends CloseContainer (or the pipeline is removed) and writing the
container's {{.container}} file fails with an {{IOException}} (EIO, ENOSPC,
EACCES) while the volume stays healthy. {{CloseContainerCommandHandler}} only
logs the error, so the container remains OPEN until the close is retried
successfully.
h3. Impact
While the container stays OPEN, the volume's committed bytes lack its headroom,
so {{AvailableSpaceFilter}} admits new containers and imports the volume has no
room for, and the storage report understates the committed space. A later
successful close or a restart corrects the counter, but containers admitted in
the meantime stay.
h3. Reproduction
[^TestBugMC8MarkForCloseRollbackReservation.java]
({{hadoop-hdds/container-service}}) uses a real {{HddsVolume}} with a mocked
1000 MB usage source, 600 MB containers, the real
{{RoundRobinVolumeChoosingPolicy}} and {{KeyValueHandler}}. The fault is a real
filesystem error: the container's metadata directory is read only (run as a non
root user). A second create is refused with {{DISK_OUT_OF_SPACE}} before the
fault. After the failed mark for close: {{state=OPEN committedSpace=false
committed=0}}, and the same second create succeeds, leaving two OPEN containers
with 1200 MB of headroom on a 1000 MB volume. A passing test means the defect
is present. Not exercised: a real EIO or ENOSPC.
h3. Patch
[^MC-8-keep-reservation-on-failed-mark-for-close.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3. Released 2.1.0 and later are affected.
{{updateContainerData}} remembers whether space was reserved before the update;
if the rollback leaves it unreserved, it calls {{ContainerData.commitSpace()}},
which reserves the current {{maxSize - bytesUsed}} again. The fix is in the
rollback because that is the only place a container legitimately returns to
OPEN, and the check of the earlier flag avoids reserving for an OPEN container
that held nothing before (which the removed unconditional {{commitSpace}} in
{{setState}} did not check). Covered by the new
{{TestKeyValueContainer.testReservationKeptWhenMarkForCloseFails}}, which also
checks that a retried mark for close then releases the space; it fails without
the change (no reservation after the rollback). With it the container, handler,
reader, persistence, close command, state machine and stale recovering scrubber
suites pass (564 tests) and checkstyle is clean.
RECOVERING containers get the same rollback once HDDS-17022 makes them release
on leaving RECOVERING and lets {{commitSpace}} reserve for them.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
