Siyao Meng created HDDS-17577:
---------------------------------
Summary: ozone admin om transfer -r can choose a listener OM as
the target and fail with "not in FOLLOWER role"
Key: HDDS-17577
URL: https://issues.apache.org/jira/browse/HDDS-17577
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-4-om-transfer-without-target-skips-listeners.patch,
TestBugCR4TransferWithoutTargetPicksListener.java
h3. Mechanism
For a transfer without a target ({{ozone admin om transfer -r}}, which sends
{{transferLeadership("")}}), {{OzoneManager.transferLeadership}} takes the
first peer of {{division.getGroup().getPeers()}} that is not the leader. Ratis
builds that group from the voters and then the listeners, but {{RaftGroup}}
keeps the peers in a {{HashMap}} keyed by peer id, so the order depends on the
node ids (their hashes and the number of peers), not on the role, and a
listener can come first. {{RatisHelper.transferRatisLeadership}} then rejects
the listener with "Target ... not in FOLLOWER role". Nothing tries another peer.
h3. Trigger
# OM HA with voters {{omNode-1}}, {{omNode-2}}, {{omNode-3}} and a listener
{{omNode-bootstrap-listener-1}} added with the normal bootstrap.
# {{omNode-3}} is the leader. For these ids the group order is {{[omNode-3,
omNode-bootstrap-listener-1, omNode-1, omNode-2]}}.
# {{ozone admin om transfer -r}} (here the same call through the client API).
Whether a cluster is affected depends on its node ids and on which voter leads.
h3. Impact
* The admin command fails every time while that leader leads, although two
voting followers are eligible. Leadership does not move.
* The failure happens before any configuration change, and a transfer with an
explicit target works.
h3. Reproduction
PASS on unmodified source. [^TestBugCR4TransferWithoutTargetPicksListener.java]
({{hadoop-ozone/integration-test}}) starts three OMs, bootstraps the listener,
moves leadership to the voter that comes first in the group's order, and
asserts that {{transferLeadership("")}} fails with "Target
omNode-bootstrap-listener-1 not in FOLLOWER role" and that the leader is
unchanged. A passing test means the defect is present.
h3. Patch
[^CR-4-om-transfer-without-target-skips-listeners.patch], against
424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
fef9cde752a.
The empty target branch now only considers peers whose startup role is
{{FOLLOWER}}, the same criterion {{RatisHelper.transferRatisLeadership}} checks
on the same group. With no voting follower besides the leader, the transfer now
fails with the existing "Cannot find a new leader to transfer leadership."
instead of the listener error (argued from code, not run).
Covered by an extension of {{testBootstrapListenerOM}} in the existing
{{TestAddRemoveOzoneManager}}, which already runs three voters and a listener:
it leads from the first voter in the group's order, asserts as a precondition
that the next peer in that order is the listener (so the test cannot pass
silently if the ids or the number of peers change the order), transfers without
a target and asserts that a voter became leader. Without the change it fails
with the "not in FOLLOWER role" error. With it {{TestAddRemoveOzoneManager}}
and {{TestTransferLeadershipShell}} (13 tests) pass, checkstyle is clean. With
the patch the reproduction fails because the transfer succeeds.
Found by code review of the OM and SCM Ratis state machine integration (leader
readiness, snapshot install and log failure handling), as part of the TLA+
verification effort under HDDS-15926, on commit
424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
