Siyao Meng created HDDS-17578:
---------------------------------
Summary: OM whose Ratis log write fails keeps running without ever
catching up or exiting, because it ignores notifyLogFailed
Key: HDDS-17578
URL: https://issues.apache.org/jira/browse/HDDS-17578
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-3-terminate-om-when-ratis-log-fails.patch,
TestBugCR3OmRatisLogFailureNoExit.java
h3. Mechanism
When a write to the Raft log fails (for example a segment roll on a volume that
turned read only), Ratis calls {{StateMachine.notifyLogFailed}} and marks the
log worker as failed. In Ratis 3.3.1 this is permanent for the process: every
later log task fails with "Log already failed", and on a leader the log is also
closed. {{SCMStateMachine.notifyLogFailed}} terminates SCM with status 1.
{{OzoneManagerStateMachine}} does not override the callback, so the OM keeps
running. It can no longer append entries, so it cannot catch up as a follower
or lead, and it never exits, so a process supervisor does not restart it,
although a restart would recover it from its intact log.
h3. Trigger
# OM HA with three OMs under client writes.
# The Raft log directory of one OM becomes read only (in the test with a
permission change, as after a read only remount).
# The next segment roll on that OM fails with {{AccessDeniedException}}; all
later appends fail.
# The directory becomes writable again; nothing changes.
For a follower this outcome is deterministic. For the leader, the OM steps down
and usually ends in the same state; in one of two runs a Ratis leader thread
first hit the closed log and the Ratis uncaught exception handler requested
exit status -1 (in production {{System.exit(-1)}}, which a shell reports as
255), which masks the problem by accident. With a single OM the same Ratis exit
request was observed during the analysis (not rerun for this report).
h3. Impact
* A failed follower always, and a failed former leader in most runs, stays up
while it is no longer part of the replicated state: its applied index stops (15
against 51 on the leader in the reproduction) and stays there after the disk is
fixed.
* Nothing signals an exit to the supervisor, so the ring silently runs with one
replica fewer. A second OM failure then stops writes although two OM processes
are running, because the dead OM still votes but cannot acknowledge appends
(argued from code, not rerun for this report).
* No data loss: the failed OM's log on disk is intact, and a restart should
recover it, catching up from the leader or from a checkpoint (argued from code,
not run).
h3. Reproduction
PASS on unmodified source. [^TestBugCR3OmRatisLogFailureNoExit.java]
({{hadoop-ozone/integration-test}}) starts three OMs with 128KB log segments,
makes a follower's Raft log directory read only, writes 40 volumes with 8KB of
metadata through the client so that the follower has to roll a segment, makes
the directory writable again, writes 10 more and waits up to 10 seconds. It
asserts that no exit was requested, the failed OM is still running and its
applied index is behind the leader's. A passing test means the defect is
present. It must not run as root.
h3. Patch
[^CR-3-terminate-om-when-ratis-log-fails.patch], against
424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
fef9cde752a.
{{OzoneManagerStateMachine.notifyLogFailed}} now calls {{ExitUtils.terminate(1,
...)}} with the failed entry (or the segment) and the cause, as SCM does and as
OM already does for fatal apply errors. Status 1 matches what HDDS-17102 asks
for abnormal OM stops. OM log entries carry no state machine data, so the other
caller of this callback (reading state machine data) does not apply to OM.
Covered by {{testNotifyLogFailedTerminates}} in the existing
{{TestOzoneManagerStateMachine}}: it asserts an exit with status 1 and the
given cause for a failed segment, and an exit with status 1 naming the term and
index for a failed entry. Without the change it fails because nothing is
thrown. With it the suite (65 tests) and {{TestOzoneManagerRatisServer}} pass,
checkstyle is clean. With the patch the reproduction fails at its first
assertion, because the failed OM requests exit status 1 as soon as the segment
roll fails. A real JVM exit was not exercised; the log worker's close waits at
most 3 seconds, so the shutdown hooks are not expected to hang (SCM uses the
same pattern).
Found by code review of the OM and SCM Ratis state machine integration (leader
readiness, snapshot install and log failure handling), as part of the TLA+
verification effort under HDDS-15926, on commit
424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
