[
https://issues.apache.org/jira/browse/HDDS-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HDDS-8770:
----------------------------------
Priority: Critical (was: Major)
> Cleanup of failed container delete may remove datanode RocksDB entries of
> active container
> ------------------------------------------------------------------------------------------
>
> Key: HDDS-8770
> URL: https://issues.apache.org/jira/browse/HDDS-8770
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Ethan Rose
> Priority: Critical
>
> Now that container schema v3 has been implemented, container level updates
> like delete and import require both moving the container directory, and
> editing the container's entries in RocksDB.
> Originally in commit bf5b6f5 the container delete steps were:
> 1. Remove entries from RocksDB
> 2. Delete container directory
> In this implementation, it is possible that the RocksDB update succeeds but
> the container delete fails, leaving behind a container directory on the disk
> that is discovered at startup. The datanode would load the container and
> recalculate only the metadata values
> (KeyValueContianerUtil#verifyAndFixupContainerData). Delete transaction and
> block data would be lost, leaving this container corrupted, but reported as
> healthy to SCM until the scanner identifies it.
> After HDDS-6449, the steps were changed so that failed directory deletes
> would not leave broken container directories that the datanode discovers on
> startup. The deletion steps became:
> 1. Move container directory to tmp deleted containers directory on the same
> file system (atomic).
> 2. Delete DB entries
> 3. Delete container from tmp directory.
> The deleted container directory will be cleared on datanode startup and
> shutdown, and this process will also clear corresponding RocksDB entries that
> may not have been cleared if an error happened after step 1. This can cause
> RocksDB data for an active container replica to be deleted incorrectly in the
> following case:
> 1. Container 1 is deleted. Rename of the container directory to the
> delete directory succeeds but DB update fails.
> 2. Container 1 is re-imported to the same datanode on the same volume.
> The imported SST files overwrite the old ones in the DB.
> 3. Datanode is restarted, triggering cleanup of the deleted container
> directory and RocksDB entries for any containers there.
> - This deletes data belonging to container ID 1, which now happens to
> belong to the active container.
> Container import can have similar issues as well. We need a standardized
> process to keep DB and directory updates consistent and recover from failures
> between the two operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
