Siyao Meng created HDDS-16839:
---------------------------------
Summary: A new OM treats any failed bootstrap reply as final and
exits, although the leader may still be adding it or may already have logged
the change
Key: HDDS-16839
URL: https://issues.apache.org/jira/browse/HDDS-16839
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-3-wait-for-membership-after-failed-bootstrap-request.patch,
TestBugCR3BootstrapFailsButChangeCommits.java
h3. Mechanism
{{OzoneManager.bootstrap}} sends one {{BootstrapOMRequest}} and ends the
process on any exception. The outcome of the request and the outcome of the
membership change are not the same thing in two cases.
# The leader answers {{success=false}} whenever
{{OzoneManagerRatisServer.addOMToRatisRing}} throws, which includes losing
leadership after the configuration entry that adds the new OM is already in the
Raft log. The next leader can still commit that entry.
# {{OMInterServiceProtocolClientSideImpl}} retries the request through the
failover proxy when the RPC itself fails (for example on
{{ozone.om.client.rpc.timeout}} or a lost connection). The first request keeps
running on the leader, so the retry is answered with {{RATIS_BOOTSTRAP_ERROR}}
"Reconfiguration is already in progress", which is not retried.
In both cases the new OM logs "Failed to Bootstrap OM." and exits. It never
looks at the Raft configuration it receives, although it is told about the
committed change through
{{OzoneManagerStateMachine.notifyConfigurationChanged}} like every other member.
h3. Trigger
Case 1 (run with timing help from the Ratis test hook, see Reproduction):
# Three OMs. The new OM is started with {{ozone om --bootstrap}} and catches up.
# The leader logs the configuration entry. One follower and the new OM are
slow, so the entry cannot be committed, the leader loses its majority and steps
down.
# The pending change is answered with a failure and the new OM exits.
# The slow follower recovers, a new leader is elected and commits the entry.
Without the hook the leader has to lose leadership or stop between appending
the first configuration entry of the change, which it does only after the new
OM has caught up, and committing the last one. That is two commit round trips,
a few milliseconds per bootstrap in a healthy ring (from the code). The test
hook holds this window open.
Case 2 (run without any hook): the first bootstrap request of the new OM times
out on the new OM while the leader is still adding it, with
{{ozone.om.client.rpc.timeout}} lowered to 200 ms in the test. The retry fails
with "Reconfiguration is already in progress" and the new OM gives up about a
quarter of a second after its first request reached the leader. With the
default timeout of 15 minutes this needs a leader that takes longer than that
to bring the new OM up to date, or a connection failure during that time
(argued, not run).
h3. Impact
* Case 1: the bootstrap command fails and the process of the new OM ends, but
the new OM is a voter in the committed Raft configuration of every existing OM.
A ring of three becomes four voters with three running, so it tolerates no
further failure until somebody notices and starts the OM. Starting it normally
(without {{--bootstrap}}) is the way out; I did not run that step.
* Case 2: a bootstrap that the leader was in the middle of completing is
reported as failed. What the leader does after the new OM is gone was not run.
h3. Reproduction
PASS on unmodified source. [^TestBugCR3BootstrapFailsButChangeCommits.java]
({{hadoop-ozone/integration-test}}) shows case 1 with the real OMs of a mini
cluster and its normal bootstrap. It uses the Ratis test hook
{{CodeInjectionForTesting}} to delay the handling of AppendEntries on one
follower and, from the entry that adds it, on the new OM. Only timing is
changed. The leader steps down with {{LOST_MAJORITY_HEARTBEATS}}, the bootstrap
fails with {{RATIS_BOOTSTRAP_ERROR}} "Failed to add OM omNode-bootstrap-1 in
Ratis ring.", and one to two seconds later, after a new leader is elected, all
three existing OMs have a stable Raft configuration with four voters that
include the new OM, whose process has ended. A passing test means the defect is
present.
Case 2 is the first regression test in the patch.
h3. Suggested fix
The attached patch changes only the new OM. It narrows the window and does not
close it.
* If the change is committed later than the wait, the new OM still exits. In
case 2 it now exits at the end of the wait instead of at once, so a leader that
finishes just then leaves the voter of case 1 (from the code, not run).
* In case 2 at the default {{ozone.om.client.rpc.timeout}} the first request
has already run for 15 minutes when the retry is refused, so the patch helps
only if the leader finishes within the wait that follows. Raising
{{ozone.om.client.rpc.timeout}} in the config of the new OM avoids the retry in
the first place (from the code, not run).
* The patch can also report success too early. The condition reads only the
state of the new OM, and it depends on the leader handing its current
configuration to a new peer before the log. Ratis 3.3.1 does that with the
snapshot notification the OM uses, once for each peer the leader starts to
bring up to date; it is a property of that implementation, not of the Ratis
API. A process that is started again with an empty disk while the leader is
still bringing the earlier process of the same OM up to date gets the log from
the start without that configuration. If its node ID was a member before and
the leader's log still has those entries, the wait can end for the old
configuration while the leader has not logged the new change. Run on a mini
cluster: the second process reported a completed bootstrap while the latest
entry in the leader's log was the old removal. Forced in that run: one log
entry per request ({{ozone.om.ratis.log.appender.queue.num-elements=1}}), the
second process started 0.6 s after the first had given up, and the old removal
held back with the Ratis test hook. Not run without those. The leader is still
adding the OM at that point, so the report is early, and wrong only if that
attempt fails afterwards.
The alternative to prefer changes the leader side.
{{OzoneManagerRatisServer.addOMToRatisRing}} adds the new OM to the peer list
without looking whether it is already a member. If it answered a repeated add
of a member with success (Ratis already answers a request that changes nothing
with success, once no change is in progress), a bounded retry in
{{OzoneManager.bootstrap}} would take the answer from the leader, and the new
OM would have nothing to infer. From the code, not run and not implemented
here. The attached patch is what the new OM can do alone.
h3. Patch
[^CR-3-wait-for-membership-after-failed-bootstrap-request.patch], against
7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
1cc6423590f (not built there).
After a failed request {{OzoneManager.bootstrap}} logs a warning with the cause
and the bound, and waits for at most
{{ozone.om.ratis.server.failure.timeout.duration}} (default 120s) until the
latest Raft configuration the new OM knows has the OM itself as a voter or
listener, has been applied locally, and is newer than the configuration the OM
had before the request and not older than any configuration it has been
notified of. If that happens the OM logs a second warning and continues its
startup, otherwise it fails as before. A bootstrap that fails for good is
therefore reported up to that much later, which is noted in the description of
the key in {{ozone-default.xml}}. No request or protocol change.
The last two parts of the condition are for a node ID that was a member before.
A new OM with an empty disk can replay old configurations that have it. The
replay starts after the last transaction of the checkpoint it gets from the
leader, so this needs a ring without writes since that membership (from the
code, not run with a write). The leader's current configuration, which Ratis
3.3.1 hands to the new OM before the log, is dropped again by Ratis when the
log is replayed in more than one request. {{OzoneManagerStateMachine}}
therefore keeps the highest configuration index it has been notified of.
Run: with a condition on the Ratis peer list of the OM, such an OM reports a
completed bootstrap for a change that is not committed. With a condition on the
latest configuration being applied alone it does so when the old log reaches it
one entry per request ({{ozone.om.ratis.log.appender.queue.num-elements=1}}).
With the default of 1024 the leader sent the old entries in one request and the
start failed correctly.
Covered in the existing {{TestAddRemoveOzoneManager}}:
* {{testBootstrapWhenRequestFailsWhileOMIsAdded}} (case 2, a voter and then a
listener). Without the change it fails with "Failed to Bootstrap OM. Error
Code: RATIS_BOOTSTRAP_ERROR, Error Message: Reconfiguration is already in
progress".
* {{testBootstrapFailsWhenOnlyOldConfigurationHasOM}}: an OM is bootstrapped,
decommissioned and bootstrapped again with an empty disk, its request fails,
and the log entries from its removal on are held back with the Ratis test hook.
The bootstrap has to fail after the wait with the original exception, while the
latest configuration of the new OM is the old one that has it.
With the change the ten tests of {{TestAddRemoveOzoneManager}} pass and
checkstyle is clean. With the patch the reproduction fails at its first
assertion because the bootstrap returns normally: the new OM logs the second
warning one to two seconds after the failed reply, once the new leader has
committed the change.
Found by code review of the OM HA membership change paths (bootstrap,
decommission and leader transfer), as part of the TLA+ verification effort
under HDDS-15926, on commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked
against HDDS issues and apache/ozone pull requests for duplicates before
filing. The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
