[
https://issues.apache.org/jira/browse/HDDS-17270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17270:
------------------------------
Description:
h3. Problem
{{OMSnapshotPurgeRequest#validateAndUpdateCache}} purges the snapshots in a
request one at a time. For each one it updates the snapshotInfoTable cache
(deep clean flags and previous snapshot IDs of the next snapshots, plus a
tombstone for the purged snapshot) and removes it from the in-memory
{{SnapshotChainManager}}. If a later snapshot in the same request throws, the
request returns an error response and none of these changes are rolled back,
while the DB batch is skipped because the response is not OK (HDDS-11452).
Most failures in that loop already terminate OM: a plain {{IOException}} maps
to {{INTERNAL_ERROR}} or {{METADATA_ERROR}}, and
{{OzoneManagerStateMachine#processResponse}} terminates OM on those, so the
partial in-memory state is discarded. The exception is {{OMException}} with
{{FILE_NOT_FOUND}} from {{SnapshotUtils#getNextSnapshot}}, thrown when the
snapshot chain has a next snapshot that is missing from snapshotInfoTable. That
status is not critical, so OM keeps running with a cache and chain that no
longer match the DB. A later write of a successor SnapshotInfo can then persist
a previous snapshot ID that skips a snapshot still present in the DB. That is
persistent snapshot chain corruption: on the next restart
{{SnapshotChainManager}} fails to load the chain and marks it corrupted.
This needs a purge request with more than one snapshot and a snapshot chain
that already disagrees with snapshotInfoTable. With a single snapshot, both
next snapshot lookups run before any change.
h3. Fix
Once the request has started changing the cache or snapshot chain, report an
{{OMException}} as {{INTERNAL_ERROR}} so that OM terminates, consistent with
how every other failure at that point is handled. On restart the chain is
rebuilt from snapshotInfoTable.
There is no protobuf, DB layout or upgrade change. The DB writes are the same
as today in every case (a failed purge writes nothing), so OMs of different
versions stay consistent. This does not make the request atomic; HDDS-11452
stays open for staging or rolling back the changes so that OM can keep running.
was:
h3. Problem
{{OMSnapshotPurgeRequest#validateAndUpdateCache}} purges the snapshots in a
request one at a time. For each one it updates the snapshotInfoTable cache
(deep clean flags and previous snapshot IDs of the next snapshots, plus a
tombstone for the purged snapshot) and removes it from the in-memory
{{SnapshotChainManager}}. If a later snapshot in the same request throws, the
request returns an error response and none of these changes are rolled back,
while the DB batch is skipped because the response is not OK (HDDS-11452).
Most failures in that loop already terminate OM: a plain {{IOException}} maps
to {{INTERNAL_ERROR}} or {{METADATA_ERROR}}, and
{{OzoneManagerStateMachine#processResponse}} terminates OM on those, so the
partial in-memory state is discarded. The exception is {{OMException}} with
{{FILE_NOT_FOUND}} from {{SnapshotUtils#getNextSnapshot}}, thrown when the
snapshot chain has a next snapshot that is missing from snapshotInfoTable. That
status is not critical, so OM keeps running with a cache and chain that no
longer match the DB. A later write of a successor SnapshotInfo can then persist
a previous snapshot ID that skips a snapshot still present in the DB.
This needs a purge request with more than one snapshot and a snapshot chain
that already disagrees with snapshotInfoTable. With a single snapshot, both
next snapshot lookups run before any change.
h3. Fix
Once the request has started changing the cache or snapshot chain, report an
{{OMException}} as {{INTERNAL_ERROR}} so that OM terminates, consistent with
how every other failure at that point is handled. On restart the chain is
rebuilt from snapshotInfoTable.
There is no protobuf, DB layout or upgrade change. The DB writes are the same
as today in every case (a failed purge writes nothing), so OMs of different
versions stay consistent. This does not make the request atomic; HDDS-11452
stays open for staging or rolling back the changes so that OM can keep running.
> Terminate OM when snapshot purge fails after partially updating the snapshot
> cache and chain
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-17270
> URL: https://issues.apache.org/jira/browse/HDDS-17270
> Project: Apache Ozone
> Issue Type: Bug
> Components: Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> {{OMSnapshotPurgeRequest#validateAndUpdateCache}} purges the snapshots in a
> request one at a time. For each one it updates the snapshotInfoTable cache
> (deep clean flags and previous snapshot IDs of the next snapshots, plus a
> tombstone for the purged snapshot) and removes it from the in-memory
> {{SnapshotChainManager}}. If a later snapshot in the same request throws, the
> request returns an error response and none of these changes are rolled back,
> while the DB batch is skipped because the response is not OK (HDDS-11452).
> Most failures in that loop already terminate OM: a plain {{IOException}} maps
> to {{INTERNAL_ERROR}} or {{METADATA_ERROR}}, and
> {{OzoneManagerStateMachine#processResponse}} terminates OM on those, so the
> partial in-memory state is discarded. The exception is {{OMException}} with
> {{FILE_NOT_FOUND}} from {{SnapshotUtils#getNextSnapshot}}, thrown when the
> snapshot chain has a next snapshot that is missing from snapshotInfoTable.
> That status is not critical, so OM keeps running with a cache and chain that
> no longer match the DB. A later write of a successor SnapshotInfo can then
> persist a previous snapshot ID that skips a snapshot still present in the DB.
> That is persistent snapshot chain corruption: on the next restart
> {{SnapshotChainManager}} fails to load the chain and marks it corrupted.
> This needs a purge request with more than one snapshot and a snapshot chain
> that already disagrees with snapshotInfoTable. With a single snapshot, both
> next snapshot lookups run before any change.
> h3. Fix
> Once the request has started changing the cache or snapshot chain, report an
> {{OMException}} as {{INTERNAL_ERROR}} so that OM terminates, consistent with
> how every other failure at that point is handled. On restart the chain is
> rebuilt from snapshotInfoTable.
> There is no protobuf, DB layout or upgrade change. The DB writes are the same
> as today in every case (a failed purge writes nothing), so OMs of different
> versions stay consistent. This does not make the request atomic; HDDS-11452
> stays open for staging or rolling back the changes so that OM can keep
> running.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
