Siyao Meng created HDDS-17548:
---------------------------------
Summary: OM client turns follower read off for the rest of its
life, or keeps retrying a not ready leader, after an OM rejects a LOCAL_LEASE
read
Key: HDDS-17548
URL: https://issues.apache.org/jira/browse/HDDS-17548
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-MC-4-local-lease-rejection-keeps-follower-read.patch,
TestBugMC1LocalLeaseRejectionDisablesFollowerRead.java,
TestBugMC4LocalLeaseNotReadyLeaderRetry.java
h3. Mechanism
{{OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOmLocalLease}}
rejects a LOCAL_LEASE read with {{createLeaderErrorException}} when local lease
is disabled on the OM or the OM fails {{allowFollowerReadLocalLease}} (last
leader RPC older than the lease time, applied index too far behind the leader's
commit index, or no known leader with a finite log limit). On a follower that
is {{OMNotLeaderException}}, on a leader that is not ready yet it is
{{OMLeaderNotReadyException}}. Both client providers,
{{HadoopRpcOMFollowerReadFailoverProxyProvider}} and
{{GrpcOMFollowerReadFailoverProxyProvider}}, treat these answers as routing
errors:
* {{OMNotLeaderException}} from the follower read node calls
{{disableFollowerRead}}. The flag is shared by the whole client and only the
constructor sets it. It is meant for a cluster whose followers do not serve
follower reads at all (for example a Ratis read option other than LINEARIZABLE,
where a LINEARIZABLE_ALLOW_FOLLOWER read gets the same exception). Afterwards
{{shouldUseFollowerRead}} is false for every request, and reads without a hint
get the leader consistency ({{ozone.client.leader.read.default.consistency}},
DEFAULT) instead of the follower consistency.
* {{OMLeaderNotReadyException}} is rethrown to the retry policy without moving
the follower read node. The policy retries the OM it believes is the leader
({{setNextOmProxy(getCurrentProxyOMNodeId())}}), and
{{selectFollowerReadNodeId}} skips that OM for a LOCAL_LEASE read, so it picks
the same not ready OM again. The rethrow assumes that the not ready OM is the
believed leader, which is not the case after an election the client has not
seen yet.
h3. Trigger
Both need a LOCAL_LEASE read:
{{ozone.client.follower.read.default.consistency=LOCAL_LEASE}}, a per request
hint through {{ClientProtocol.setThreadLocalReadConsistency}}, or the S3 header
{{x-ozone-read-consistency: follower-stale}} (with an optional
{{x-ozone-local-lease-log-limit}}). The S3 gateway honours the header only when
security is off; with security on it sends every S3 request as
LINEARIZABLE_LEADER_ONLY. Both providers are always installed, and a read with
an explicit LOCAL_LEASE hint goes to a follower even with
{{ozone.client.follower.read.enabled=false}} (the default). The consistency
change in the first impact below needs
{{ozone.client.follower.read.enabled=true}} with LINEARIZABLE_ALLOW_FOLLOWER as
the follower default; with follower read off on the client, unhinted reads
already go to the leader as DEFAULT, and a rejected hinted read only stops
later hinted reads from using followers.
* Follower read turned off: one LOCAL_LEASE read reaches a follower that fails
the lease check. At default settings that happens during an election, after a
gap of more than 5 s in leader RPCs, or more than 10000 entries behind the
leader. With a small per request log limit (0 from the S3 header) a follower
that is one entry behind under write load is enough. An OM with
{{ozone.om.follower.read.local.lease.enabled=false}} (the default) rejects
every LOCAL_LEASE read this way.
* Retries pinned to a not ready leader: a new leader is elected before the
client learns about it (it learns from the {{OMNotLeaderException}} redirect of
a request sent to the old leader). The new leader rejects LOCAL_LEASE reads
while it is not ready and its apply backlog is larger than the log limit (10000
by default, or the per request limit). The old leader, now a follower, and the
third OM pass the same check and could serve.
h3. Impact
* Follower read stays off for the rest of the client's life. The S3 gateway
uses one {{OzoneClient}} for all requests, so a single request can turn it off
for every user of that gateway until it restarts. All reads then go to the
leader. Reads without a hint that the client would send as
LINEARIZABLE_ALLOW_FOLLOWER (the default follower consistency) are sent as
DEFAULT, which the leader serves locally after checking its own Raft role,
without a ReadIndex round. That is what a client without follower read gets,
but it is weaker than the configured linearizable read: a leader that has lost
leadership without noticing yet can return stale data (argued from the code,
not shown). For a client whose default is LOCAL_LEASE the reads get stronger,
and the effect is only the loss of follower read. On a cluster with local lease
disabled, LOCAL_LEASE reads also switch off the LINEARIZABLE_ALLOW_FOLLOWER
follower reads the cluster does serve.
* The LOCAL_LEASE read is retried only on the not ready OM, with the growing
wait of the retry policy, until that OM is ready or
{{ozone.client.failover.max.attempts}} is used up, while the other OMs could
serve it at once. At default settings (500 attempts, 2 s base wait) this is
mostly a stall that can last past the not ready window; a failed read needs a
smaller budget or a long window. Another request on the same client that
corrects the believed leader ends it. Since HDDS-17448 the gRPC client (the S3
gateway path) does the same as the Hadoop RPC client; before it, the gRPC
client fell back to the leader here.
h3. Reproduction
[^TestBugMC1LocalLeaseRejectionDisablesFollowerRead.java] runs in
{{hadoop-ozone/integration-test}} on the shared three OM cluster of
{{OzoneManagerHAFollowerReadTests}} (Ratis LINEARIZABLE read, local lease
enabled), with the Hadoop RPC and the gRPC transport, through the client API
plus the existing test hooks that pin the follower read node
({{changeInitialProxyForTest}}, {{changeLeaderProxyForTest}},
{{changeFollowerReadInitialProxy}}). One LOCAL_LEASE read with a per request
lease time of 0 ms is rejected by the follower (its
{{NumFollowerReadLocalLeaseFailTime}} metric goes up) and served by the leader.
Afterwards 5 reads without a hint, pointed at the same follower, did 0
linearizable reads on the follower and 0 on the leader with both transports.
With the patch the follower served all 5 as linearizable reads.
[^TestBugMC4LocalLeaseNotReadyLeaderRetry.java] runs in
{{hadoop-ozone/common}}. The Hadoop RPC retry proxy and the gRPC transport run
unmodified against three simulated OMs that answer like
{{submitReadRequestToOmLocalLease}} does for their role: a write teaches the
client that om1 is the leader, then om2 becomes the leader and stays not ready.
With a budget of 6 failovers the LOCAL_LEASE read went to om2 on every attempt
(7 with Hadoop RPC, 6 with gRPC) and failed, and om1 and om3 got no request.
With the patch it is served at once. The pinned retries were not produced on a
real cluster: the existing test hooks cannot hold a newly elected OM not ready
with a backlog above the limit.
h3. Patch
[^MC-1-MC-4-local-lease-rejection-keeps-follower-read.patch], against
efe0c7eba1527530d1001f6517b733576fdc6d61, which is also current master.
In both providers, an {{OMNotLeaderException}} or {{OMLeaderNotReadyException}}
answer to a LOCAL_LEASE read now moves the follower read node to the next OM
and sends this read to the leader, as the {{OMNotLeaderException}} branch
already did, but without {{disableFollowerRead}}. Other reads keep the current
handling: a cluster that does not serve LINEARIZABLE_ALLOW_FOLLOWER still turns
follower read off
({{TestOzoneManagerHAWithAllRunning.testOMFollowerReadWithClusterDisabled}}),
and a not ready leader is still retried after a wait (HDDS-17448). There is no
server, protocol or configuration change. A separate server change makes OMs
reject more LOCAL_LEASE reads; this client fix should land first or together
with it. One behaviour changes for a setup without the bug: a LOCAL_LEASE
client against OMs with local lease disabled, or against OMs older than
follower read support (2.1.x and older answer every read on a follower with
{{OMNotLeaderException}}, which matters when clients are upgraded first), now
sends each LOCAL_LEASE read to one follower and then to the leader (one extra
RPC per read), where before the first rejection sent all later reads to the
leader. That cost is acceptable because the rejection is a cheap role and
configuration check on the OM, is logged at DEBUG only, and only hits clients
that asked for LOCAL_LEASE, while the old behaviour silently changed the read
consistency of every later read. Trying the remaining followers before the
leader, as for {{ReadIndexException}}, would cost up to one RPC per OM per read
in that setup, with a warning per failed follower on the Hadoop RPC path, so
the patch goes to the leader right away.
Covered by {{testLocalLeaseRejectionKeepsFollowerRead}} and
{{testLocalLeaseReadOnNotReadyLeaderUnknownToClient}} in
{{TestHadoopRpcOMFollowerReadFailoverProxyProvider}} (through the retry proxy)
and {{testLocalLeaseRejectionFallsBackToLeader}} (explicit hint) and
{{testLocalLeaseRejectionWithLocalLeaseDefault}} (no hint, client default
LOCAL_LEASE) for both exceptions in
{{TestGrpcOMFollowerReadFailoverProxyProvider}}. Without the change they fail
with "expected: <true> but was: <false>" for the follower read flag and with
the {{OMLeaderNotReadyException}} thrown to the caller. With it the client side
follower read and failover suites in {{hadoop-ozone/common}} pass (98 tests),
as do {{TestOzoneManagerHAFollowerReadWithAllRunning}},
{{TestOzoneManagerHAFollowerReadWithStoppedNodes}},
{{TestOzoneShellHAWithFollowerRead}} and
{{testOMFollowerReadWithClusterDisabled}} (26 tests), and checkstyle is clean.
Found by TLA+ model checking and code review of the OM follower read and read
consistency routing paths under HDDS-15926, on commit
efe0c7eba1527530d1001f6517b733576fdc6d61. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16435 (a superseded leader serving leader local reads, which
is the consistency the first symptom falls back to), HDDS-17448 (resolved,
retry of a not ready leader that is the believed leader) and HDDS-14509
(resolved, added the client read consistency levels). The attached patch is a
proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
