[
https://issues.apache.org/jira/browse/HDDS-17013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned HDDS-17013:
---------------------------------
Assignee: Andy Huang
> Tarball and untar directory of a container import interrupted by a datanode
> crash are never removed from the volume
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17013
> URL: https://issues.apache.org/jira/browse/HDDS-17013
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Andy Huang
> Priority: Major
> Attachments:
> CR-16-remove-interrupted-import-leftovers-at-startup.patch,
> TestBugCR16StaleImportTarball.java
>
>
> h3. Mechanism
> A pushed container is written by {{SendContainerRequestHandler}} to
> {{<volume>/tmp/container-copy/container-<id>-<random UUID>.tar}}, then
> untarred into {{container-copy/<id>}} by
> {{TarContainerPacker.unpackContainerData}}, called from
> {{KeyValueContainer.importContainerData}}. The tarball is deleted only in
> process ({{onError}}, the failure path of {{onCompleted}}, the {{finally}} of
> {{ContainerImporter.importContainer}}); the untar directory is deleted by the
> failure path of {{KeyValueContainer.importContainerData}} or moved into place
> on success. Nothing else lists or cleans {{container-copy}}: the volume
> initialization ({{HddsVolume.createTmpDirs}}) cleans only directories under
> {{hdds/<clusterID>/tmp}}, container loading at startup does not look at it,
> and a graceful shutdown does not either. A later import of the same container
> to the same volume replaces the untar directory, but because the tarball name
> contains a random UUID it never replaces the old tarball.
> Released versions are affected as well (checked 2.2.1): there the pull
> replication path also writes {{container-<id>-<UUID>.tar}} into the same
> directory.
> h3. Trigger
> # SCM has a container pushed to a datanode, and the datanode has received
> part of it (or is untarring it).
> # The datanode process dies before the import finishes (kill, OOM kill or
> exit on OOM, power loss).
> # The datanode restarts. The leftovers stay.
> h3. Impact
> Each import in flight at a crash leaves up to about the size of the container
> on that volume (more if the untar had started) until an operator deletes it
> by hand. The loss accumulates: a datanode that crashes repeatedly while
> receiving re-replication, for example during a decommission or after another
> datanode failed, adds more on each crash, and nothing in Ozone reports where
> the space went. The space is outside the container accounting, so it lowers
> the free space the datanode reports (argued from the code, not measured). No
> data loss.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugCR16StaleImportTarball.java]
> ({{hadoop-hdds/container-service}}) models a datanode lifetime by what
> startup does per data volume ({{StorageVolumeUtil.checkVolume}}, then
> {{ContainerReader}} as in {{OzoneContainer.buildContainerSet}}). It pushes
> three 1MB parts of a container through the real
> {{SendContainerRequestHandler}} and abandons the handler without {{onError}}
> or {{onCompleted}}, as after a kill. The tarball is still there after a
> restart, after a failed push of the same container, and after a volume set
> shutdown and a second restart. A passing test means the defect is present.
> h3. Patch
> [^CR-16-remove-interrupted-import-leftovers-at-startup.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> {{ContainerReader}} empties the volume's {{container-copy}} directory before
> it loads the containers of the volume at datanode startup (the
> {{shouldDelete}} path; the debug CLI does not use it). Nothing can be
> importing at that point, since the replication server is started later in
> {{OzoneContainer.start}}. The cleanup is not done in
> {{HddsVolume.createTmpDirs}}, because that runs on every version handshake
> with SCM, including after a reregister, when an import may be in progress. An
> exception during the cleanup only logs a warning and does not stop the volume
> from loading. The cleanup also removes files left in that directory by older
> versions after an upgrade.
> Covered by {{testLeftoverContainerImportFilesCleared}} in the existing
> {{TestContainerReader}}, which also checks that a run without
> {{shouldDelete}} leaves the files alone. Without the change it fails for all
> container test versions with the directory not empty; with it the replication
> and container loading suites ({{TestSendContainerRequestHandler}},
> {{TestContainerImporter}}, {{TestGrpcReplicationService}},
> {{TestContainerReader}}, {{TestOzoneContainer}}, 139 tests) pass and
> checkstyle is clean. With the patch the reproduction fails because the
> tarball is gone after the first restart.
> HDDS-8162 (under HDDS-8161) would move {{container-copy}} under
> {{hdds/<clusterID>/tmp}}, which HDDS-8161 proposes to clean on restart; this
> patch does not move the directory and does not conflict with that.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
