[
https://issues.apache.org/jira/browse/HDDS-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17737:
------------------------------
Attachment: MC-10-legacy-diff-job-not-recorded-ordered.patch
> Ordered snapshot diff served by the legacy diff is stored as ordered, so a
> later ordered request never replaces it
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17737
> URL: https://issues.apache.org/jira/browse/HDDS-17737
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-10-legacy-diff-job-not-recorded-ordered.patch,
> TestBugMC10LegacyFallbackRecordedOrdered.java
>
>
> h3. Mechanism
> {{SnapshotDiffManager.submitSnapshotDiff}} stores the job's ordered flag as
> "optimized diff enabled and ordered requested". The job then runs the legacy
> diff, which does not order the report, whenever the optimized diff is enabled
> but the native SST diff is unavailable (native library not loaded or
> {{ozone.om.snapshot.diff.disable.native.libs}}) and a full diff is not
> forced. That routing is decided from the same request flags and server
> settings, so it is known at submit time.
> A QUEUED or DONE previous job is replaced by a new request only if {{isFso &&
> effectiveOrderedReport && !snapDiffJob.isOrderedReport()}}, that is, only if
> the stored job is not ordered. Because the legacy job was stored as ordered,
> a later ordered request that would run the optimized diff (for example with a
> forced full diff) gets the legacy report back until the job expires.
> h3. Trigger
> Optimized snapshot diff enabled
> ({{ozone.om.snapshot.diff.optimized.enabled}}, default false), native SST
> diff unavailable, full diff not forced on the server, an FSO bucket.
> # An ordered snapshot diff request without a forced full diff. It runs the
> legacy diff.
> # Another ordered request for the same pair with a forced full diff.
> h3. Impact
> The second request is answered "Previous snapshot diff attempt found. Status:
> DONE." with the legacy, type ordered report, for example {{DELETE dir1/dir2}}
> before {{RENAME dir1/dir2/key1 -> dir1/dir3/renamed_key}}, which does not
> replay in that order. The response reason is empty, so the client is not told
> the report is ordered, but it cannot get the ordered report for that pair
> until the job expires. The job records an ordered request that the legacy
> diff never attempts, so the resubmit check treats the legacy report as an
> ordered one.
> h3. Reproduction
> [^TestBugMC10LegacyFallbackRecordedOrdered.java] is a self contained mini
> cluster class (target path and Maven command in the class comment) with the
> optimized diff enabled, {{ozone.om.snapshot.diff.disable.native.libs=true}}
> and full diff not forced on the server, using only public client calls.
> History: {{dir1/dir2/key1, dir1/dir3/key1; snapshot 1; mv dir1/dir2/key1
> dir1/dir3/renamed_key; rm -r dir1/dir2; snapshot 2}}. On unmodified source
> the first ordered request ends DONE with an empty reason and {{DELETE
> dir1/dir2, MODIFY dir1/dir3, RENAME dir1/dir2/key1 ->
> dir1/dir3/renamed_key}}, and the second ordered request with a forced full
> diff returns the same job and report. A passing test means the defect is
> present.
> h3. Patch
> [^MC-10-legacy-diff-job-not-recorded-ordered.patch], against
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
> bbc8505e00d.
> The existing routing condition moves into a private helper
> {{SnapshotDiffManager.isLegacyDiff}}, used by {{generateSnapshotDiffReport}}
> (unchanged behaviour) and by {{submitSnapshotDiff}}, which now stores the job
> as ordered only if the optimized diff will serve it, and logs that the
> ordered request is ignored otherwise (the existing log for a disabled
> optimized diff, widened). The job's ordered field now means the optimized
> diff attempts ordering, and a one line comment says so where it is computed;
> whether the report is ordered is in the DONE reason, as for the over the
> limit fallback. A legacy job is then stored like a request made while the
> optimized diff is disabled. A later ordered request that runs the optimized
> diff passes the existing resubmit check; one that would run the legacy diff
> again does not, so there is no recompute loop. Storing the outcome after the
> run instead was not taken, because then every ordered request with the same
> flags would rerun the legacy diff.
> Covered by {{testSubmitOrderedServedByLegacyDiffIsNotRecordedOrdered}} in the
> existing {{TestSnapshotDiffManager}}. Behaviour change visible in two
> existing tests: {{testSubmitDoneResubmitsFsoOrdered}} and
> {{testSubmitQueuedResubmitsFsoOrdered}} passed before although the unit test
> JVM has no native library, so their requests were served by the legacy diff:
> an ordered request served by the legacy diff replaced an unordered job and
> reran the legacy diff. That no longer happens, so they now force a full diff
> to keep testing the optimized resubmit. Without the change the new test fails
> with "expected: <false> but was: <true>"; with it
> {{TestSnapshotDiffManager}}, {{TestSnapshotDiffManagerMXBean}},
> {{TestSnapDiffDependencyGraph}} and {{TestMergeJoinSnapDiffWriter}} pass (123
> tests) and checkstyle is clean. With the patch the reproduction fails as
> intended: the forced request is resubmitted ("Previous snapshot diff attempt
> found with an unordered report") and ends DONE "Report is dependency
> ordered." with {{RENAME dir1/dir2/key1 -> dir1/dir3/renamed_key, DELETE
> dir1/dir2}}. Jobs already stored with the flag expire as before.
> Found by TLA+ model checking and code review of the OM optimized snapshot
> diff paths under HDDS-15926, on commit
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
