[
https://issues.apache.org/jira/browse/HDDS-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17579:
------------------------------
Attachment: MC-2-new-om-leader-ready-only-after-previous-term-applied.patch
> New OM leader reports ready while previous term entries are still unapplied,
> so default reads miss acknowledged writes
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17579
> URL: https://issues.apache.org/jira/browse/HDDS-17579
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-new-om-leader-ready-only-after-previous-term-applied.patch,
> TestBugMC2NewLeaderStaleRead.java
>
>
> h3. Mechanism
> {{OzoneManagerRatisServer.getLeaderStatus}} returns {{LEADER_AND_READY}} as
> soon as Ratis reports the division as leader ready. Ratis (3.3.1) sets that
> when its state machine updater reaches the leader's startup configuration
> entry ({{LeaderStateImpl.checkReady}}), and the updater counts an entry as
> applied once {{applyTransaction}} has returned its future.
> {{OzoneManagerStateMachine.applyTransaction}} only queues the entry on the
> single OM apply executor ({{CompletableFuture.supplyAsync}}), and
> {{notifyLeaderReady}} does not wait for that queue. So a new leader can be
> ready while entries from the previous term, including writes already
> acknowledged to their clients, are still queued and not in the OM table cache
> or DB.
> On {{LEADER_AND_READY}} the default read path
> ({{OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOmDefault}})
> and the leader local branches of the other read paths answer from local
> state. Background services that check {{OzoneManager.isLeaderReady}} start in
> the same window (HDDS-16795 is one consequence).
> h3. Trigger
> Leadership moves to an OM whose apply executor has a backlog, or an OM
> restarts with committed entries that were not flushed before it stopped, and
> a read arrives before the backlog is applied.
> # Leader change (run): in a 3 OM ring, a follower's apply executor is held up
> (in the test by holding a bucket lock on that follower). A key create in that
> bucket, then a create of {{key1}} in another bucket, are acknowledged by the
> leader and replicated. Leadership is transferred to the lagging follower.
> Ratis logs "is ready since appliedIndex == startIndex", {{getLeaderStatus}}
> returns {{LEADER_AND_READY}}, and a client {{getKey(key1)}} returns
> {{KEY_NOT_FOUND}}, {{listKeys}} returns nothing. After the queue drains the
> key is there.
> # Single OM restart (run during the analysis, not part of the attached test):
> an OM stopped after acknowledging writes but before the double buffer flushed
> them replays them in a new term after restart, and gives the same result. The
> unflushed stop was simulated by pausing the double buffer before the stop;
> the replay backlog was natural in one run (1500 key commits, Ratis ready at
> index 4507 while OM had applied up to 1953) and came from a held bucket lock
> in the other.
> The window lasts until the executor has caught up, at most
> {{ozone.om.unflushed.transaction.max.count}} (default 10000) entries behind.
> In the leader change case, a follower that keeps up has no backlog and is not
> affected.
> h3. Impact
> * A client reading through the default path (and the leader local branches of
> the follower read paths) gets {{KEY_NOT_FOUND}}, an empty listing or other
> stale metadata for objects whose write was already acknowledged. The answer
> is final for the client; nothing retries or corrects it.
> * Services gated by {{isLeaderReady}} (key and directory deletion, open key
> and multipart cleanup, lifecycle, Ranger sync, upgrade finalization) can run
> against state that is missing committed entries.
> * Linearizable reads served by the new leader through Ratis are affected the
> same way. Ratis 3.3.1 ({{StateMachineUpdater.applyLog}}) passes a
> configuration entry's index to the ReadIndex wait as applied right away,
> while the futures of earlier OM entries can still be pending, and with no new
> writes the read index is the startup index. On unmodified source with
> {{ozone.om.ratis.server.read.option=LINEARIZABLE}},
> {{ozone.om.allow.leader.skip.linearizable.read=false}} and the leader lease
> off, reads of {{key1}} with {{LINEARIZABLE_LEADER_ONLY}} and with
> {{LINEARIZABLE_ALLOW_FOLLOWER}}, both served by the new leader through Ratis,
> returned {{KEY_NOT_FOUND}} (run during the review of this report).
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC2NewLeaderStaleRead.java]
> ({{hadoop-ozone/integration-test}}) runs a 3 OM mini cluster, holds the
> bucket lock of {{bucket2}} on a follower, writes a key in {{bucket2}} and
> {{key1}} in {{bucket1}} through the client, transfers leadership to that
> follower with the Ratis admin API and reads {{key1}} through the client. It
> asserts {{LEADER_AND_READY}}, {{KEY_NOT_FOUND}} and an empty listing, then
> that {{key1}} is readable once the lock is released. A passing test means the
> defect is present. The held lock stands in for a slow apply; the restart
> variant was shown earlier with a natural replay backlog.
> h3. Patch
> [^MC-2-new-om-leader-ready-only-after-previous-term-applied.patch], against
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
> fef9cde752a.
> {{getLeaderStatus}} now also requires that the term of the OM state machine's
> last applied index equals the current term. OM advances that index after the
> double buffer has flushed an entry, and moves it onto a following Ratis
> configuration or metadata entry once everything before it is flushed. Every
> term starts with the leader's startup configuration entry, so the index is in
> the current term exactly when everything committed in earlier terms is
> applied and flushed. A leader with no backlog is not held back: the index
> moves onto the startup entry right after Ratis applies it. With a backlog the
> leader reports {{LEADER_AND_NOT_READY}} and clients retry. Writes are
> rejected during the backlog too, so clients retry after
> {{ozone.client.wait.between.retries.millis}} instead of queueing behind it.
> That also keeps {{createClientRequest}} and {{preExecute}} from running
> against the same stale local state, where a write could fail with
> {{BUCKET_NOT_FOUND}} for a bucket created just before the leader change, or
> be built for the wrong bucket layout (argued from code, not run). No new
> state or config.
> On the new leader this closes the no hint read path, {{DEFAULT}} and
> {{LINEARIZABLE_LEADER_ONLY}}, which now reject with
> {{OMLeaderNotReadyException}} until the backlog is applied (with the patch, a
> {{LINEARIZABLE_LEADER_ONLY}} read kept retrying while the backlog was held,
> run during the review). It does not close {{LINEARIZABLE_ALLOW_FOLLOWER}}
> landing on a not ready leader, which still goes to the Ratis ReadIndex wait
> described above (argued from code), {{LOCAL_LEASE}} reads on a not ready
> leader, covered by HDDS-17550, or linearizable reads on a follower, whose own
> ReadIndex wait passes configuration entries in the same way (argued from
> code). Rejecting {{LEADER_AND_NOT_READY}} in the
> {{LINEARIZABLE_ALLOW_FOLLOWER}} branch, as the {{LINEARIZABLE_LEADER_ONLY}}
> branch already does, is a possible follow up.
> Related: this is not a duplicate of HDDS-16795, which reports double purges
> and quota drift in {{KeyDeletingService}} with the same root cause.
> {{KeyDeletingService}} and {{DirectoryDeletingService}} check
> {{isLeaderReady}} before both the active object store and the snapshot
> branches, so this patch also closes the window of HDDS-16795 for runs that
> start after the leader change or restart, including the snapshot path, and
> the separate check proposed there is not needed if this change lands.
> HDDS-16794 (a run already in progress across the change) is not covered.
> RATIS-2313 asked Ratis to wait for pending state machine futures before
> marking the leader ready and was resolved Not A Bug: a state machine that
> needs this should wait itself, so the check belongs in Ozone. HDDS-13357
> records the same assumption for {{resetInFlightSnapshotCount}}. Before Ozone
> 1.4.0, Ratis 2.x readiness required the state machine's own applied index
> ({{LeaderStateImpl.isReady}}); the window opened with the move to Ratis 3
> (HDDS-9192), and this patch restores that meaning in OM. HDDS-10546 made OM
> move that index onto the following configuration entries once earlier entries
> are flushed, which this check relies on.
> Covered by {{testNewLeaderNotReadyUntilPreviousTermApplied}} in the existing
> {{TestOzoneManagerHAWithStoppedNodes}}: the same leader change with a held
> lock, asserting {{LEADER_AND_NOT_READY}} while Ratis already reports the
> division ready, and that the key is readable once the backlog drains. Without
> the change to {{OzoneManagerRatisServer}} it fails with "expected:
> LEADER_AND_NOT_READY but was: LEADER_AND_READY". With it,
> {{TestOzoneManagerHAWithStoppedNodes}} and
> {{TestOzoneManagerHAWithAllRunning}} (44 tests), the OM state machine, Ratis
> server, double buffer, background service and key manager unit suites (566
> tests), {{TestTransferLeadershipShell}}, {{TestAddRemoveOzoneManager}},
> {{TestOzoneManagerHAFollowerReadWithAllRunning}}, {{TestOzoneManagerRestart}}
> and {{TestOMUpgradeFinalization}} (34 tests) pass, checkstyle is clean. With
> the patch the reproduction fails at its {{LEADER_AND_READY}} assertion,
> because the new leader reports {{LEADER_AND_NOT_READY}} until the backlog is
> applied. The single OM restart variant was not rerun with the patch.
> Found by TLA+ model checking and code review of the OM and SCM Ratis state
> machine integration (leader readiness, snapshot install and log failure
> handling) under HDDS-15926, on commit
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
