[
https://issues.apache.org/jira/browse/HDDS-17741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17741:
------------------------------
Attachment: MC-7-replace-job-and-queue-old-report-in-one-write.patch
> OM crash during an ordered snapshot diff resubmit can leave a DONE job whose
> report the cleanup then deletes
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17741
> URL: https://issues.apache.org/jira/browse/HDDS-17741
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-7-replace-job-and-queue-old-report-in-one-write.patch,
> TestBugMC7CrashDuringResubmitLosesDoneReport.java
>
>
> h3. Mechanism
> With {{ozone.om.snapshot.diff.optimized.enabled}} = true (default false), on
> an FSO bucket, an ordered request for a pair whose job is DONE (or QUEUED)
> with an unordered report replaces that job.
> {{SnapshotDiffManager.createSnapDiffJob}} does this with two separate RocksDB
> writes: {{enqueueSnapshotDiffReportCleanup}} puts the old job ID into the
> purged job table, then the new job is put into the job table. If the OM
> process dies between the two writes, or the second write fails without a
> crash (the case the regression test simulates), the old job is still the DONE
> job of its key while its report is queued for deletion. The next
> {{SnapshotDiffCleanupService}} run deletes the report rows of every queued
> job ID without looking at the job table, and {{loadJobsOnStartUp}} only
> handles IN_PROGRESS jobs.
> h3. Trigger
> # An unordered snapshot diff of a pair finishes DONE.
> # The same pair is submitted with an ordered report.
> # The OM process dies after the first write of the resubmit and before the
> second.
> # After the restart, a cleanup run happens before the job expires.
> h3. Impact
> The job stays DONE but its report is gone. Every read of it fails, at the
> moment with the NullPointerException of HDDS-16442 in
> {{checkReportsIntegrity}}, until the job expires (default 7 days) or someone
> submits an ordered request again. With HDDS-16442 fixed, the first read would
> mark the job FAILED and return an error, and the next submit of the pair
> would recompute it (argued from the code, not run). Only the snapshot diff
> report is affected.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC7CrashDuringResubmitLosesDoneReport.java]
> ({{hadoop-ozone/ozone-manager}}) runs the real {{SnapshotDiffManager}} and
> {{SnapshotDiffCleanupService}} on a real RocksDB with the rest of the OM
> mocked. A stand in for the report generation task writes three report rows
> and publishes DONE through the real {{updateJobStatusToDone}}. The crash is
> modelled by failing the serialization of the new job, which happens after the
> first write and before the second; the restart is a new
> {{SnapshotDiffManager}} on the same DB. A real process kill was not run.
> Output on d3b75c75bf9ad475551202477c3a21a3f4b968b7: the old job is queued for
> purge after the crash, and after the restart and one cleanup run the job is
> still DONE with 0 of 3 report rows, and reading it throws. A passing test
> means the defect is present.
> h3. Patch
> [^MC-7-replace-job-and-queue-old-report-in-one-write.patch], against
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
> bbc8505e00d.
> When a previous job exists, {{createSnapDiffJob}} now writes the purged table
> entry of the old job and the new job in one {{ManagedWriteBatch}}
> ({{replaceSnapDiffJob}}), so either both are durable or neither is. A first
> submission still uses a plain put. A failed write now fails the resubmit and
> leaves the old job as it was, where it used to be logged while the resubmit
> went on.
> Covered by {{testResubmitQueuesReplacedJobForCleanupOnlyWithItsReplacement}}
> in the existing {{TestSnapshotDiffManager}}: with the write of the new job
> failing, the ordered resubmit must leave the DONE job and no purged entry;
> without the failure it must replace the job and queue the old one. Without
> the change the first part fails because the old job is queued. With it
> {{TestSnapshotDiffManager}}, {{TestSnapshotDiffCleanupService}} and
> {{TestSnapshotDiffManagerMXBean}} (88 tests) pass, checkstyle is clean, and
> the reproduction fails because all three rows remain and the read succeeds.
> Found by TLA+ model checking and code review of the OM optimized snapshot
> diff paths under HDDS-15926, on commit
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
