[
https://issues.apache.org/jira/browse/HDDS-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17548:
------------------------------
Attachment: TestBugMC4LocalLeaseNotReadyLeaderRetry.java
> OM client turns follower read off for the rest of its life, or keeps retrying
> a not ready leader, after an OM rejects a LOCAL_LEASE read
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17548
> URL: https://issues.apache.org/jira/browse/HDDS-17548
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-MC-4-local-lease-rejection-keeps-follower-read.patch,
> TestBugMC1LocalLeaseRejectionDisablesFollowerRead.java,
> TestBugMC4LocalLeaseNotReadyLeaderRetry.java
>
>
> h3. Mechanism
> {{OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOmLocalLease}}
> rejects a LOCAL_LEASE read with {{createLeaderErrorException}} when local
> lease is disabled on the OM or the OM fails {{allowFollowerReadLocalLease}}
> (last leader RPC older than the lease time, applied index too far behind the
> leader's commit index, or no known leader with a finite log limit). On a
> follower that is {{OMNotLeaderException}}, on a leader that is not ready yet
> it is {{OMLeaderNotReadyException}}. Both client providers,
> {{HadoopRpcOMFollowerReadFailoverProxyProvider}} and
> {{GrpcOMFollowerReadFailoverProxyProvider}}, treat these answers as routing
> errors:
> * {{OMNotLeaderException}} from the follower read node calls
> {{disableFollowerRead}}. The flag is shared by the whole client and only the
> constructor sets it. It is meant for a cluster whose followers do not serve
> follower reads at all (for example a Ratis read option other than
> LINEARIZABLE, where a LINEARIZABLE_ALLOW_FOLLOWER read gets the same
> exception). Afterwards {{shouldUseFollowerRead}} is false for every request,
> and reads without a hint get the leader consistency
> ({{ozone.client.leader.read.default.consistency}}, DEFAULT) instead of the
> follower consistency.
> * {{OMLeaderNotReadyException}} is rethrown to the retry policy without
> moving the follower read node. The policy retries the OM it believes is the
> leader ({{setNextOmProxy(getCurrentProxyOMNodeId())}}), and
> {{selectFollowerReadNodeId}} skips that OM for a LOCAL_LEASE read, so it
> picks the same not ready OM again. The rethrow assumes that the not ready OM
> is the believed leader, which is not the case after an election the client
> has not seen yet.
> h3. Trigger
> Both need a LOCAL_LEASE read:
> {{ozone.client.follower.read.default.consistency=LOCAL_LEASE}}, a per request
> hint through {{ClientProtocol.setThreadLocalReadConsistency}}, or the S3
> header {{x-ozone-read-consistency: follower-stale}} (with an optional
> {{x-ozone-local-lease-log-limit}}). The S3 gateway honours the header only
> when security is off; with security on it sends every S3 request as
> LINEARIZABLE_LEADER_ONLY. Both providers are always installed, and a read
> with an explicit LOCAL_LEASE hint goes to a follower even with
> {{ozone.client.follower.read.enabled=false}} (the default). The consistency
> change in the first impact below needs
> {{ozone.client.follower.read.enabled=true}} with LINEARIZABLE_ALLOW_FOLLOWER
> as the follower default; with follower read off on the client, unhinted reads
> already go to the leader as DEFAULT, and a rejected hinted read only stops
> later hinted reads from using followers.
> * Follower read turned off: one LOCAL_LEASE read reaches a follower that
> fails the lease check. At default settings that happens during an election,
> after a gap of more than 5 s in leader RPCs, or more than 10000 entries
> behind the leader. With a small per request log limit (0 from the S3 header)
> a follower that is one entry behind under write load is enough. An OM with
> {{ozone.om.follower.read.local.lease.enabled=false}} (the default) rejects
> every LOCAL_LEASE read this way.
> * Retries pinned to a not ready leader: a new leader is elected before the
> client learns about it (it learns from the {{OMNotLeaderException}} redirect
> of a request sent to the old leader). The new leader rejects LOCAL_LEASE
> reads while it is not ready and its apply backlog is larger than the log
> limit (10000 by default, or the per request limit). The old leader, now a
> follower, and the third OM pass the same check and could serve.
> h3. Impact
> * Follower read stays off for the rest of the client's life. The S3 gateway
> uses one {{OzoneClient}} for all requests, so a single request can turn it
> off for every user of that gateway until it restarts. All reads then go to
> the leader. Reads without a hint that the client would send as
> LINEARIZABLE_ALLOW_FOLLOWER (the default follower consistency) are sent as
> DEFAULT, which the leader serves locally after checking its own Raft role,
> without a ReadIndex round. That is what a client without follower read gets,
> but it is weaker than the configured linearizable read: a leader that has
> lost leadership without noticing yet can return stale data (argued from the
> code, not shown). For a client whose default is LOCAL_LEASE the reads get
> stronger, and the effect is only the loss of follower read. On a cluster with
> local lease disabled, LOCAL_LEASE reads also switch off the
> LINEARIZABLE_ALLOW_FOLLOWER follower reads the cluster does serve.
> * The LOCAL_LEASE read is retried only on the not ready OM, with the growing
> wait of the retry policy, until that OM is ready or
> {{ozone.client.failover.max.attempts}} is used up, while the other OMs could
> serve it at once. At default settings (500 attempts, 2 s base wait) this is
> mostly a stall that can last past the not ready window; a failed read needs a
> smaller budget or a long window. Another request on the same client that
> corrects the believed leader ends it. Since HDDS-17448 the gRPC client (the
> S3 gateway path) does the same as the Hadoop RPC client; before it, the gRPC
> client fell back to the leader here.
> h3. Reproduction
> [^TestBugMC1LocalLeaseRejectionDisablesFollowerRead.java] runs in
> {{hadoop-ozone/integration-test}} on the shared three OM cluster of
> {{OzoneManagerHAFollowerReadTests}} (Ratis LINEARIZABLE read, local lease
> enabled), with the Hadoop RPC and the gRPC transport, through the client API
> plus the existing test hooks that pin the follower read node
> ({{changeInitialProxyForTest}}, {{changeLeaderProxyForTest}},
> {{changeFollowerReadInitialProxy}}). One LOCAL_LEASE read with a per request
> lease time of 0 ms is rejected by the follower (its
> {{NumFollowerReadLocalLeaseFailTime}} metric goes up) and served by the
> leader. Afterwards 5 reads without a hint, pointed at the same follower, did
> 0 linearizable reads on the follower and 0 on the leader with both
> transports. With the patch the follower served all 5 as linearizable reads.
> [^TestBugMC4LocalLeaseNotReadyLeaderRetry.java] runs in
> {{hadoop-ozone/common}}. The Hadoop RPC retry proxy and the gRPC transport
> run unmodified against three simulated OMs that answer like
> {{submitReadRequestToOmLocalLease}} does for their role: a write teaches the
> client that om1 is the leader, then om2 becomes the leader and stays not
> ready. With a budget of 6 failovers the LOCAL_LEASE read went to om2 on every
> attempt (7 with Hadoop RPC, 6 with gRPC) and failed, and om1 and om3 got no
> request. With the patch it is served at once. The pinned retries were not
> produced on a real cluster: the existing test hooks cannot hold a newly
> elected OM not ready with a backlog above the limit.
> h3. Patch
> [^MC-1-MC-4-local-lease-rejection-keeps-follower-read.patch], against
> efe0c7eba1527530d1001f6517b733576fdc6d61, which is also current master.
> In both providers, an {{OMNotLeaderException}} or
> {{OMLeaderNotReadyException}} answer to a LOCAL_LEASE read now moves the
> follower read node to the next OM and sends this read to the leader, as the
> {{OMNotLeaderException}} branch already did, but without
> {{disableFollowerRead}}. Other reads keep the current handling: a cluster
> that does not serve LINEARIZABLE_ALLOW_FOLLOWER still turns follower read off
> ({{TestOzoneManagerHAWithAllRunning.testOMFollowerReadWithClusterDisabled}}),
> and a not ready leader is still retried after a wait (HDDS-17448). There is
> no server, protocol or configuration change. A separate server change makes
> OMs reject more LOCAL_LEASE reads; this client fix should land first or
> together with it. One behaviour changes for a setup without the bug: a
> LOCAL_LEASE client against OMs with local lease disabled, or against OMs
> older than follower read support (2.1.x and older answer every read on a
> follower with {{OMNotLeaderException}}, which matters when clients are
> upgraded first), now sends each LOCAL_LEASE read to one follower and then to
> the leader (one extra RPC per read), where before the first rejection sent
> all later reads to the leader. That cost is acceptable because the rejection
> is a cheap role and configuration check on the OM, is logged at DEBUG only,
> and only hits clients that asked for LOCAL_LEASE, while the old behaviour
> silently changed the read consistency of every later read. Trying the
> remaining followers before the leader, as for {{ReadIndexException}}, would
> cost up to one RPC per OM per read in that setup, with a warning per failed
> follower on the Hadoop RPC path, so the patch goes to the leader right away.
> Covered by {{testLocalLeaseRejectionKeepsFollowerRead}} and
> {{testLocalLeaseReadOnNotReadyLeaderUnknownToClient}} in
> {{TestHadoopRpcOMFollowerReadFailoverProxyProvider}} (through the retry
> proxy) and {{testLocalLeaseRejectionFallsBackToLeader}} (explicit hint) and
> {{testLocalLeaseRejectionWithLocalLeaseDefault}} (no hint, client default
> LOCAL_LEASE) for both exceptions in
> {{TestGrpcOMFollowerReadFailoverProxyProvider}}. Without the change they fail
> with "expected: <true> but was: <false>" for the follower read flag and with
> the {{OMLeaderNotReadyException}} thrown to the caller. With it the client
> side follower read and failover suites in {{hadoop-ozone/common}} pass (98
> tests), as do {{TestOzoneManagerHAFollowerReadWithAllRunning}},
> {{TestOzoneManagerHAFollowerReadWithStoppedNodes}},
> {{TestOzoneShellHAWithFollowerRead}} and
> {{testOMFollowerReadWithClusterDisabled}} (26 tests), and checkstyle is clean.
> Found by TLA+ model checking and code review of the OM follower read and read
> consistency routing paths under HDDS-15926, on commit
> efe0c7eba1527530d1001f6517b733576fdc6d61. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-16435 (a superseded leader serving leader local reads, which
> is the consistency the first symptom falls back to), HDDS-17448 (resolved,
> retry of a not ready leader that is the believed leader) and HDDS-14509
> (resolved, added the client read consistency levels). The attached patch is a
> proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
