[
https://issues.apache.org/jira/browse/HDDS-17026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17026:
------------------------------
Attachment: TestBugMC10DbLoadReservationLeak.java
> Datanode container create that fails with an unchecked exception leaks the
> reserved container size on the volume until restart
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17026
> URL: https://issues.apache.org/jira/browse/HDDS-17026
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-10-release-reservation-on-unchecked-create-failure.patch,
> TestBugMC10DbLoadReservationLeak.java
>
>
> h3. Mechanism
> {{KeyValueContainer.create}} reserves the container size ({{maxSize}}, 5 GB
> by default) on the volume that {{chooseVolume}} picks.
> {{KeyValueHandler.handleCreateContainer}} gives that reservation back only in
> its {{catch (StorageContainerException)}} branch. When {{create}} fails with
> an unchecked exception after the volume was chosen, the exception passes that
> branch, {{KeyValueHandler.handle}} turns it into a
> {{CONTAINER_INTERNAL_ERROR}} response, and the volume's committed bytes stay
> raised by the full container size. Nothing recomputes committed bytes while
> the datanode runs, so each such attempt adds another {{maxSize}} until
> restart. Present since HDDS-12810 (2.1.0), which moved the reservation into
> {{chooseVolume}}; 2.0.0 is not affected (checked in the ozone-2.0.0, 2.1.0
> and 2.2.1 tags).
> h3. Trigger
> Today this is reached through the state described in HDDS-11485: a data
> volume whose per-volume DB did not load at restart (for example its
> {{DS-<id>}} directory or {{container.db}} is gone, or a dedicated DB volume
> was lost) stays in service with {{dbParentDir}} unset. Every container create
> placed on it, including the implicit create {{HddsDispatcher}} issues for the
> first write to a new container, throws {{NullPointerException("dbParentDir ==
> null")}} from {{KeyValueContainerLocationUtil.getContainerDBFile}} after the
> reservation. Any other unchecked failure in {{create}} after the volume is
> chosen leaks the same way.
> h3. Impact
> * Each failed create leaves {{maxSize}} committed on the volume. In the
> reproduction three creates left 15 GB committed on an empty volume, and the
> storage report sent to SCM carries it.
> * The volume choosing policy counts committed bytes against the volume's free
> space, so the phantom bytes grow until they fill it; after that the policy
> stops picking the volume (by code reading, not run). On the known trigger
> every create on that volume fails anyway, so the harm is the wrong committed
> and free space figures, on the datanode and in what SCM sees.
> * The creates themselves fail either way; this report is only about the space
> they leave reserved.
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC10DbLoadReservationLeak.java]
> ({{hadoop-hdds/container-service}}, target path and Maven command in the
> class comment) formats a data volume whose DB lives on a dedicated DB volume,
> loses the DB disk and drops it from the configuration, then rebuilds the
> volume set with a real {{StorageVolumeChecker}}, loads the DBs and runs the
> startup volume check. The volume is {{NORMAL}} with {{isDbLoaded=false}} and
> {{dbParentDir=null}}. Three CreateContainer requests go through
> {{KeyValueHandler.handle}}; each returns {{CONTAINER_INTERNAL_ERROR}} with
> "dbParentDir == null", no container is added, and committed bytes grow to
> 5368709120, 10737418240 and 16106127360. A later volume check still finds the
> volume healthy and its report carries 3 x 5 GB committed. A passing test
> means the defect is present.
> h3. Patch
> [^MC-10-release-reservation-on-unchecked-create-failure.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> {{handleCreateContainer}} now releases the reservation in its {{finally}}
> block whenever the container was not created, instead of only in the
> {{StorageContainerException}} branch. {{ContainerData.releaseCommitSpace}},
> unchanged, already gives back the space only while the reservation is still
> held and then clears the flag, so a path that already released it is not
> released twice, and the branch for an already existing container, which
> reserves nothing, is unaffected. Volume state handling is not changed: with
> the patch such a volume still stays in service and the creates still fail,
> but nothing stays reserved. The root cause of that volume state is tracked in
> HDDS-11485; this fix is independent of it.
> Covered by {{testCreateContainerReleasesSpaceOnUncheckedFailure}} in the
> existing {{TestKeyValueHandler}}: a schema V3 volume with its working
> directory but no per-volume DB, one CreateContainer through
> {{KeyValueHandler.handle}}. Without the change it fails with "expected: 0L
> but was: 5368709120L"; with it the volume has no committed bytes. The new
> {{testCreateContainerKeepsSpaceOfCreatedContainer}} covers the guard: a
> successful create keeps {{maxSize}} committed with the reservation flag set,
> and a repeated create of the same container leaves both unchanged; it fails
> if the release is made unconditional. The existing
> {{testCreateContainerWithFailure}}, which makes {{ContainerSet.addContainer}}
> throw {{StorageContainerException}}, covers the checked path and passes
> unchanged. With the change the handler, container, dispatcher, persistence
> and {{OzoneContainer}} suites (551 tests) pass and checkstyle is clean. With
> the patch the reproduction fails at its committed bytes check after the first
> create, while the volume is still {{NORMAL}}.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-11485 (the volume state that reaches this today) and
> HDDS-17009 (releases the reservation on a volume that the retry loop in
> {{KeyValueContainer.create}} gives up on; the two patches are independent and
> apply together). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
