[
https://issues.apache.org/jira/browse/HDDS-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17021:
------------------------------
Attachment: CR-17-keep-recovering-container-tracked-on-failed-close.patch
> Failed mark for close of a RECOVERING container drops it from stale
> recovering tracking, so it is never marked UNHEALTHY
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17021
> URL: https://issues.apache.org/jira/browse/HDDS-17021
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-17-keep-recovering-container-tracked-on-failed-close.patch,
> TestBugCR17FailedCloseUntracksRecoveringContainer.java
>
>
> h3. Mechanism
> {{KeyValueHandler.markContainerForClose}} removes a RECOVERING container from
> the stale recovering tracking map
> ({{ContainerSet.removeRecoveringContainer}}) before it calls
> {{KeyValueContainer.markContainerForClose}}. When writing the {{.container}}
> file then fails, {{KeyValueContainer.updateContainerData}} rolls the state
> back to RECOVERING, but the tracking entry is not restored.
> {{StaleRecoveringContainerScrubbingService}} only looks at that map, so it
> never marks the container UNHEALTHY, and {{ContainerSet.getContainerReport}}
> skips RECOVERING containers, so SCM never sees it either.
> h3. Trigger
> # EC reconstruction creates a RECOVERING target container and writes to it.
> # The coordinator's {{CloseContainer}} fails on that target because the
> {{.container}} file cannot be written (EIO, ENOSPC, EACCES) while the volume
> stays healthy.
> # The coordinator's cleanup ({{deleteContainerInState}}) does not remove the
> container.
> In the usual case step 3 does not happen:
> {{ECReconstructionCoordinator.reconstructECContainerGroup}} deletes every
> target it created after any failure. From code reading (not run), it does not
> when the delete RPC fails (the error is only logged), when the coordinator,
> which is another target, dies before the delete, or when {{deleteInternal}}
> gives up on the container lock and returns success without deleting. No other
> close path reaches a RECOVERING container.
> h3. Impact
> The container stays RECOVERING in memory, untracked and unreported, until the
> datanode restarts ({{ContainerReader}} then marks it UNHEALTHY or deletes
> it). Until then a later reconstruction of the same container that picks this
> datanode as a target fails at container creation
> ({{CONTAINER_ALREADY_EXISTS}}), because SCM does not know the replica is
> there. It also holds its data on disk, and SCM cannot remove it. Without the
> bug the scrubber marks it UNHEALTHY {{ozone.recovering.container.timeout}}
> (default 20m) after its last write, after which it is reported and can be
> removed. No data loss. Affects all releases since 1.3.0 (HDDS-6987).
> h3. Reproduction
> PASS on unmodified source, deterministic.
> [^TestBugCR17FailedCloseUntracksRecoveringContainer.java]
> ({{hadoop-hdds/container-service}}) creates two RECOVERING containers through
> {{KeyValueHandler.handleCreateContainer}} on a real {{HddsVolume}} with a
> recovering timeout of 0. The first is closed through {{handleCloseContainer}}
> while its metadata directory is read only (run as a non root user), the
> second is the control; then one scrubber run is executed. Output: {{failed
> close: result=CONTAINER_FILES_CREATE_ERROR state=RECOVERING
> inRecoveringMap=false}} and {{after scrub: control state=UNHEALTHY, failed
> state=RECOVERING, containers in report=[2]}}. A passing test means the defect
> is present. Not exercised: a real EIO or ENOSPC, the reconstruction
> coordinator, SCM.
> h3. Patch
> [^CR-17-keep-recovering-container-tracked-on-failed-close.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> The tracking entry is now removed only after the state change to CLOSING
> succeeded, under the same container write lock. On failure the entry stays
> with its original deadline, so the scrubber still marks the container
> UNHEALTHY in time. Restoring the entry on failure was not chosen, because the
> existing helper resets the deadline. The {{logRecovered}} container log line
> is unchanged. On a successful close the entry now stays in the map during the
> {{.container}} write, so a container whose deadline has already passed can be
> picked up by a scrubber run in that window and later marked UNHEALTHY
> although it closed. That is the existing HDDS-16455 race widened by one file
> write, and its proposed fix (check the state under the container lock in the
> scrub task) covers it. Covered by the new
> {{TestKeyValueHandler.testFailedCloseKeepsRecoveringContainerTracked}}, which
> fails without the change (empty tracking map after the failed close) and also
> checks that a successful close drops the entry. With the change the handler,
> scrubber, container, persistence, reader, container set and close command
> suites pass and checkstyle is clean. With the patch the reproduction fails,
> because the container is tracked and marked UNHEALTHY like the control.
> Related, not duplicates: HDDS-16460 (stale entry left after delete),
> HDDS-16455 and HDDS-16456 (races between the scrubber, the timeout update and
> a successful close).
> Found by code review of the datanode volume and per-disk RocksDB lifecycle
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
