Siyao Meng created HDDS-17021:
---------------------------------
Summary: Failed mark for close of a RECOVERING container drops it
from stale recovering tracking, so it is never marked UNHEALTHY
Key: HDDS-17021
URL: https://issues.apache.org/jira/browse/HDDS-17021
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-17-keep-recovering-container-tracked-on-failed-close.patch,
TestBugCR17FailedCloseUntracksRecoveringContainer.java
h3. Mechanism
{{KeyValueHandler.markContainerForClose}} removes a RECOVERING container from
the stale recovering tracking map ({{ContainerSet.removeRecoveringContainer}})
before it calls {{KeyValueContainer.markContainerForClose}}. When writing the
{{.container}} file then fails, {{KeyValueContainer.updateContainerData}} rolls
the state back to RECOVERING, but the tracking entry is not restored.
{{StaleRecoveringContainerScrubbingService}} only looks at that map, so it
never marks the container UNHEALTHY, and {{ContainerSet.getContainerReport}}
skips RECOVERING containers, so SCM never sees it either.
h3. Trigger
# EC reconstruction creates a RECOVERING target container and writes to it.
# The coordinator's {{CloseContainer}} fails on that target because the
{{.container}} file cannot be written (EIO, ENOSPC, EACCES) while the volume
stays healthy.
# The coordinator's cleanup ({{deleteContainerInState}}) does not remove the
container.
In the usual case step 3 does not happen:
{{ECReconstructionCoordinator.reconstructECContainerGroup}} deletes every
target it created after any failure. From code reading (not run), it does not
when the delete RPC fails (the error is only logged), when the coordinator,
which is another target, dies before the delete, or when {{deleteInternal}}
gives up on the container lock and returns success without deleting. No other
close path reaches a RECOVERING container.
h3. Impact
The container stays RECOVERING in memory, untracked and unreported, until the
datanode restarts ({{ContainerReader}} then marks it UNHEALTHY or deletes it).
Until then a later reconstruction of the same container that picks this
datanode as a target fails at container creation
({{CONTAINER_ALREADY_EXISTS}}), because SCM does not know the replica is there.
It also holds its data on disk, and SCM cannot remove it. Without the bug the
scrubber marks it UNHEALTHY {{ozone.recovering.container.timeout}} (default
20m) after its last write, after which it is reported and can be removed. No
data loss. Affects all releases since 1.3.0 (HDDS-6987).
h3. Reproduction
PASS on unmodified source, deterministic.
[^TestBugCR17FailedCloseUntracksRecoveringContainer.java]
({{hadoop-hdds/container-service}}) creates two RECOVERING containers through
{{KeyValueHandler.handleCreateContainer}} on a real {{HddsVolume}} with a
recovering timeout of 0. The first is closed through {{handleCloseContainer}}
while its metadata directory is read only (run as a non root user), the second
is the control; then one scrubber run is executed. Output: {{failed close:
result=CONTAINER_FILES_CREATE_ERROR state=RECOVERING inRecoveringMap=false}}
and {{after scrub: control state=UNHEALTHY, failed state=RECOVERING, containers
in report=[2]}}. A passing test means the defect is present. Not exercised: a
real EIO or ENOSPC, the reconstruction coordinator, SCM.
h3. Patch
[^CR-17-keep-recovering-container-tracked-on-failed-close.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
The tracking entry is now removed only after the state change to CLOSING
succeeded, under the same container write lock. On failure the entry stays with
its original deadline, so the scrubber still marks the container UNHEALTHY in
time. Restoring the entry on failure was not chosen, because the existing
helper resets the deadline. The {{logRecovered}} container log line is
unchanged. On a successful close the entry now stays in the map during the
{{.container}} write, so a container whose deadline has already passed can be
picked up by a scrubber run in that window and later marked UNHEALTHY although
it closed. That is the existing HDDS-16455 race widened by one file write, and
its proposed fix (check the state under the container lock in the scrub task)
covers it. Covered by the new
{{TestKeyValueHandler.testFailedCloseKeepsRecoveringContainerTracked}}, which
fails without the change (empty tracking map after the failed close) and also
checks that a successful close drops the entry. With the change the handler,
scrubber, container, persistence, reader, container set and close command
suites pass and checkstyle is clean. With the patch the reproduction fails,
because the container is tracked and marked UNHEALTHY like the control.
Related, not duplicates: HDDS-16460 (stale entry left after delete), HDDS-16455
and HDDS-16456 (races between the scrubber, the timeout update and a successful
close).
Found by code review of the datanode volume and per-disk RocksDB lifecycle
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
