Siyao Meng created HDDS-17270:
---------------------------------
Summary: Terminate OM when snapshot purge fails after partially
updating the snapshot cache and chain
Key: HDDS-17270
URL: https://issues.apache.org/jira/browse/HDDS-17270
Project: Apache Ozone
Issue Type: Bug
Components: Snapshot
Reporter: Siyao Meng
h3. Problem
{{OMSnapshotPurgeRequest#validateAndUpdateCache}} purges the snapshots in a
request one at a time. For each one it updates the snapshotInfoTable cache
(deep clean flags and previous snapshot IDs of the next snapshots, plus a
tombstone for the purged snapshot) and removes it from the in-memory
{{SnapshotChainManager}}. If a later snapshot in the same request throws, the
request returns an error response and none of these changes are rolled back,
while the DB batch is skipped because the response is not OK (HDDS-11452).
Most failures in that loop already terminate OM: a plain {{IOException}} maps
to {{INTERNAL_ERROR}} or {{METADATA_ERROR}}, and
{{OzoneManagerStateMachine#processResponse}} terminates OM on those, so the
partial in-memory state is discarded. The exception is {{OMException}} with
{{FILE_NOT_FOUND}} from {{SnapshotUtils#getNextSnapshot}}, thrown when the
snapshot chain has a next snapshot that is missing from snapshotInfoTable. That
status is not critical, so OM keeps running with a cache and chain that no
longer match the DB. A later write of a successor SnapshotInfo can then persist
a previous snapshot ID that skips a snapshot still present in the DB.
This needs a purge request with more than one snapshot and a snapshot chain
that already disagrees with snapshotInfoTable. With a single snapshot, both
next snapshot lookups run before any change.
h3. Fix
Once the request has started changing the cache or snapshot chain, report an
{{OMException}} as {{INTERNAL_ERROR}} so that OM terminates, consistent with
how every other failure at that point is handled. On restart the chain is
rebuilt from snapshotInfoTable.
There is no protobuf, DB layout or upgrade change. The DB writes are the same
as today in every case (a failed purge writes nothing), so OMs of different
versions stay consistent. This does not make the request atomic; HDDS-11452
stays open for staging or rolling back the changes so that OM can keep running.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
