Siyao Meng created HDDS-17579:
---------------------------------
Summary: New OM leader reports ready while previous term entries
are still unapplied, so default reads miss acknowledged writes
Key: HDDS-17579
URL: https://issues.apache.org/jira/browse/HDDS-17579
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-new-om-leader-ready-only-after-previous-term-applied.patch,
TestBugMC2NewLeaderStaleRead.java
h3. Mechanism
{{OzoneManagerRatisServer.getLeaderStatus}} returns {{LEADER_AND_READY}} as
soon as Ratis reports the division as leader ready. Ratis (3.3.1) sets that
when its state machine updater reaches the leader's startup configuration entry
({{LeaderStateImpl.checkReady}}), and the updater counts an entry as applied
once {{applyTransaction}} has returned its future.
{{OzoneManagerStateMachine.applyTransaction}} only queues the entry on the
single OM apply executor ({{CompletableFuture.supplyAsync}}), and
{{notifyLeaderReady}} does not wait for that queue. So a new leader can be
ready while entries from the previous term, including writes already
acknowledged to their clients, are still queued and not in the OM table cache
or DB.
On {{LEADER_AND_READY}} the default read path
({{OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOmDefault}})
and the leader local branches of the other read paths answer from local state.
Background services that check {{OzoneManager.isLeaderReady}} start in the same
window (HDDS-16795 is one consequence).
h3. Trigger
Leadership moves to an OM whose apply executor has a backlog, or an OM restarts
with committed entries that were not flushed before it stopped, and a read
arrives before the backlog is applied.
# Leader change (run): in a 3 OM ring, a follower's apply executor is held up
(in the test by holding a bucket lock on that follower). A key create in that
bucket, then a create of {{key1}} in another bucket, are acknowledged by the
leader and replicated. Leadership is transferred to the lagging follower. Ratis
logs "is ready since appliedIndex == startIndex", {{getLeaderStatus}} returns
{{LEADER_AND_READY}}, and a client {{getKey(key1)}} returns {{KEY_NOT_FOUND}},
{{listKeys}} returns nothing. After the queue drains the key is there.
# Single OM restart (run during the analysis, not part of the attached test):
an OM stopped after acknowledging writes but before the double buffer flushed
them replays them in a new term after restart, and gives the same result. The
unflushed stop was simulated by pausing the double buffer before the stop; the
replay backlog was natural in one run (1500 key commits, Ratis ready at index
4507 while OM had applied up to 1953) and came from a held bucket lock in the
other.
The window lasts until the executor has caught up, at most
{{ozone.om.unflushed.transaction.max.count}} (default 10000) entries behind. In
the leader change case, a follower that keeps up has no backlog and is not
affected.
h3. Impact
* A client reading through the default path (and the leader local branches of
the follower read paths) gets {{KEY_NOT_FOUND}}, an empty listing or other
stale metadata for objects whose write was already acknowledged. The answer is
final for the client; nothing retries or corrects it.
* Services gated by {{isLeaderReady}} (key and directory deletion, open key and
multipart cleanup, lifecycle, Ranger sync, upgrade finalization) can run
against state that is missing committed entries.
* Linearizable reads served by the new leader through Ratis are affected the
same way. Ratis 3.3.1 ({{StateMachineUpdater.applyLog}}) passes a configuration
entry's index to the ReadIndex wait as applied right away, while the futures of
earlier OM entries can still be pending, and with no new writes the read index
is the startup index. On unmodified source with
{{ozone.om.ratis.server.read.option=LINEARIZABLE}},
{{ozone.om.allow.leader.skip.linearizable.read=false}} and the leader lease
off, reads of {{key1}} with {{LINEARIZABLE_LEADER_ONLY}} and with
{{LINEARIZABLE_ALLOW_FOLLOWER}}, both served by the new leader through Ratis,
returned {{KEY_NOT_FOUND}} (run during the review of this report).
h3. Reproduction
PASS on unmodified source. [^TestBugMC2NewLeaderStaleRead.java]
({{hadoop-ozone/integration-test}}) runs a 3 OM mini cluster, holds the bucket
lock of {{bucket2}} on a follower, writes a key in {{bucket2}} and {{key1}} in
{{bucket1}} through the client, transfers leadership to that follower with the
Ratis admin API and reads {{key1}} through the client. It asserts
{{LEADER_AND_READY}}, {{KEY_NOT_FOUND}} and an empty listing, then that
{{key1}} is readable once the lock is released. A passing test means the defect
is present. The held lock stands in for a slow apply; the restart variant was
shown earlier with a natural replay backlog.
h3. Patch
[^MC-2-new-om-leader-ready-only-after-previous-term-applied.patch], against
424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
fef9cde752a.
{{getLeaderStatus}} now also requires that the term of the OM state machine's
last applied index equals the current term. OM advances that index after the
double buffer has flushed an entry, and moves it onto a following Ratis
configuration or metadata entry once everything before it is flushed. Every
term starts with the leader's startup configuration entry, so the index is in
the current term exactly when everything committed in earlier terms is applied
and flushed. A leader with no backlog is not held back: the index moves onto
the startup entry right after Ratis applies it. With a backlog the leader
reports {{LEADER_AND_NOT_READY}} and clients retry. Writes are rejected during
the backlog too, so clients retry after
{{ozone.client.wait.between.retries.millis}} instead of queueing behind it.
That also keeps {{createClientRequest}} and {{preExecute}} from running against
the same stale local state, where a write could fail with {{BUCKET_NOT_FOUND}}
for a bucket created just before the leader change, or be built for the wrong
bucket layout (argued from code, not run). No new state or config.
On the new leader this closes the no hint read path, {{DEFAULT}} and
{{LINEARIZABLE_LEADER_ONLY}}, which now reject with
{{OMLeaderNotReadyException}} until the backlog is applied (with the patch, a
{{LINEARIZABLE_LEADER_ONLY}} read kept retrying while the backlog was held, run
during the review). It does not close {{LINEARIZABLE_ALLOW_FOLLOWER}} landing
on a not ready leader, which still goes to the Ratis ReadIndex wait described
above (argued from code), {{LOCAL_LEASE}} reads on a not ready leader, covered
by HDDS-17550, or linearizable reads on a follower, whose own ReadIndex wait
passes configuration entries in the same way (argued from code). Rejecting
{{LEADER_AND_NOT_READY}} in the {{LINEARIZABLE_ALLOW_FOLLOWER}} branch, as the
{{LINEARIZABLE_LEADER_ONLY}} branch already does, is a possible follow up.
Related: this is not a duplicate of HDDS-16795, which reports double purges and
quota drift in {{KeyDeletingService}} with the same root cause.
{{KeyDeletingService}} and {{DirectoryDeletingService}} check {{isLeaderReady}}
before both the active object store and the snapshot branches, so this patch
also closes the window of HDDS-16795 for runs that start after the leader
change or restart, including the snapshot path, and the separate check proposed
there is not needed if this change lands. HDDS-16794 (a run already in progress
across the change) is not covered. RATIS-2313 asked Ratis to wait for pending
state machine futures before marking the leader ready and was resolved Not A
Bug: a state machine that needs this should wait itself, so the check belongs
in Ozone. HDDS-13357 records the same assumption for
{{resetInFlightSnapshotCount}}. Before Ozone 1.4.0, Ratis 2.x readiness
required the state machine's own applied index ({{LeaderStateImpl.isReady}});
the window opened with the move to Ratis 3 (HDDS-9192), and this patch restores
that meaning in OM. HDDS-10546 made OM move that index onto the following
configuration entries once earlier entries are flushed, which this check relies
on.
Covered by {{testNewLeaderNotReadyUntilPreviousTermApplied}} in the existing
{{TestOzoneManagerHAWithStoppedNodes}}: the same leader change with a held
lock, asserting {{LEADER_AND_NOT_READY}} while Ratis already reports the
division ready, and that the key is readable once the backlog drains. Without
the change to {{OzoneManagerRatisServer}} it fails with "expected:
LEADER_AND_NOT_READY but was: LEADER_AND_READY". With it,
{{TestOzoneManagerHAWithStoppedNodes}} and {{TestOzoneManagerHAWithAllRunning}}
(44 tests), the OM state machine, Ratis server, double buffer, background
service and key manager unit suites (566 tests),
{{TestTransferLeadershipShell}}, {{TestAddRemoveOzoneManager}},
{{TestOzoneManagerHAFollowerReadWithAllRunning}}, {{TestOzoneManagerRestart}}
and {{TestOMUpgradeFinalization}} (34 tests) pass, checkstyle is clean. With
the patch the reproduction fails at its {{LEADER_AND_READY}} assertion, because
the new leader reports {{LEADER_AND_NOT_READY}} until the backlog is applied.
The single OM restart variant was not rerun with the patch.
Found by TLA+ model checking and code review of the OM and SCM Ratis state
machine integration (leader readiness, snapshot install and log failure
handling) under HDDS-15926, on commit 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
