Siyao Meng created HDDS-17741:
---------------------------------
Summary: OM crash during an ordered snapshot diff resubmit can
leave a DONE job whose report the cleanup then deletes
Key: HDDS-17741
URL: https://issues.apache.org/jira/browse/HDDS-17741
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-7-replace-job-and-queue-old-report-in-one-write.patch,
TestBugMC7CrashDuringResubmitLosesDoneReport.java
h3. Mechanism
With {{ozone.om.snapshot.diff.optimized.enabled}} = true (default false), on an
FSO bucket, an ordered request for a pair whose job is DONE (or QUEUED) with an
unordered report replaces that job. {{SnapshotDiffManager.createSnapDiffJob}}
does this with two separate RocksDB writes:
{{enqueueSnapshotDiffReportCleanup}} puts the old job ID into the purged job
table, then the new job is put into the job table. If the OM process dies
between the two writes, or the second write fails without a crash (the case the
regression test simulates), the old job is still the DONE job of its key while
its report is queued for deletion. The next {{SnapshotDiffCleanupService}} run
deletes the report rows of every queued job ID without looking at the job
table, and {{loadJobsOnStartUp}} only handles IN_PROGRESS jobs.
h3. Trigger
# An unordered snapshot diff of a pair finishes DONE.
# The same pair is submitted with an ordered report.
# The OM process dies after the first write of the resubmit and before the
second.
# After the restart, a cleanup run happens before the job expires.
h3. Impact
The job stays DONE but its report is gone. Every read of it fails, at the
moment with the NullPointerException of HDDS-16442 in
{{checkReportsIntegrity}}, until the job expires (default 7 days) or someone
submits an ordered request again. With HDDS-16442 fixed, the first read would
mark the job FAILED and return an error, and the next submit of the pair would
recompute it (argued from the code, not run). Only the snapshot diff report is
affected.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC7CrashDuringResubmitLosesDoneReport.java]
({{hadoop-ozone/ozone-manager}}) runs the real {{SnapshotDiffManager}} and
{{SnapshotDiffCleanupService}} on a real RocksDB with the rest of the OM
mocked. A stand in for the report generation task writes three report rows and
publishes DONE through the real {{updateJobStatusToDone}}. The crash is
modelled by failing the serialization of the new job, which happens after the
first write and before the second; the restart is a new {{SnapshotDiffManager}}
on the same DB. A real process kill was not run. Output on
d3b75c75bf9ad475551202477c3a21a3f4b968b7: the old job is queued for purge after
the crash, and after the restart and one cleanup run the job is still DONE with
0 of 3 report rows, and reading it throws. A passing test means the defect is
present.
h3. Patch
[^MC-7-replace-job-and-queue-old-report-in-one-write.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
When a previous job exists, {{createSnapDiffJob}} now writes the purged table
entry of the old job and the new job in one {{ManagedWriteBatch}}
({{replaceSnapDiffJob}}), so either both are durable or neither is. A first
submission still uses a plain put. A failed write now fails the resubmit and
leaves the old job as it was, where it used to be logged while the resubmit
went on.
Covered by {{testResubmitQueuesReplacedJobForCleanupOnlyWithItsReplacement}} in
the existing {{TestSnapshotDiffManager}}: with the write of the new job
failing, the ordered resubmit must leave the DONE job and no purged entry;
without the failure it must replace the job and queue the old one. Without the
change the first part fails because the old job is queued. With it
{{TestSnapshotDiffManager}}, {{TestSnapshotDiffCleanupService}} and
{{TestSnapshotDiffManagerMXBean}} (88 tests) pass, checkstyle is clean, and the
reproduction fails because all three rows remain and the read succeeds.
Found by TLA+ model checking and code review of the OM optimized snapshot diff
paths under HDDS-15926, on commit d3b75c75bf9ad475551202477c3a21a3f4b968b7.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
