[
https://issues.apache.org/jira/browse/HDDS-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17578:
------------------------------
Attachment: TestBugCR3OmRatisLogFailureNoExit.java
> OM whose Ratis log write fails keeps running without ever catching up or
> exiting, because it ignores notifyLogFailed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17578
> URL: https://issues.apache.org/jira/browse/HDDS-17578
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 0.5.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0,
> 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-3-terminate-om-when-ratis-log-fails.patch,
> TestBugCR3OmRatisLogFailureNoExit.java
>
>
> h3. Mechanism
> When a write to the Raft log fails (for example a segment roll on a volume
> that turned read only), Ratis calls {{StateMachine.notifyLogFailed}} and
> marks the log worker as failed. In Ratis 3.3.1 this is permanent for the
> process: every later log task fails with "Log already failed", and on a
> leader the log is also closed. {{SCMStateMachine.notifyLogFailed}} terminates
> SCM with status 1. {{OzoneManagerStateMachine}} does not override the
> callback, so the OM keeps running. It can no longer append entries, so it
> cannot catch up as a follower or lead, and it never exits, so a process
> supervisor does not restart it, although a restart would recover it from its
> intact log.
> h3. Trigger
> # OM HA with three OMs under client writes.
> # The Raft log directory of one OM becomes read only (in the test with a
> permission change, as after a read only remount).
> # The next segment roll on that OM fails with {{AccessDeniedException}}; all
> later appends fail.
> # The directory becomes writable again; nothing changes.
> For a follower this outcome is deterministic. For the leader, the OM steps
> down and usually ends in the same state; in one of two runs a Ratis leader
> thread first hit the closed log and the Ratis uncaught exception handler
> requested exit status -1 (in production {{System.exit(-1)}}, which a shell
> reports as 255), which masks the problem by accident. With a single OM the
> same Ratis exit request was observed during the analysis (not rerun for this
> report).
> h3. Impact
> * A failed follower always, and a failed former leader in most runs, stays up
> while it is no longer part of the replicated state: its applied index stops
> (15 against 51 on the leader in the reproduction) and stays there after the
> disk is fixed.
> * Nothing signals an exit to the supervisor, so the ring silently runs with
> one replica fewer. A second OM failure then stops writes although two OM
> processes are running, because the dead OM still votes but cannot acknowledge
> appends (argued from code, not rerun for this report).
> * No data loss: the failed OM's log on disk is intact, and a restart should
> recover it, catching up from the leader or from a checkpoint (argued from
> code, not run).
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR3OmRatisLogFailureNoExit.java]
> ({{hadoop-ozone/integration-test}}) starts three OMs with 128KB log segments,
> makes a follower's Raft log directory read only, writes 40 volumes with 8KB
> of metadata through the client so that the follower has to roll a segment,
> makes the directory writable again, writes 10 more and waits up to 10
> seconds. It asserts that no exit was requested, the failed OM is still
> running and its applied index is behind the leader's. A passing test means
> the defect is present. It must not run as root.
> h3. Patch
> [^CR-3-terminate-om-when-ratis-log-fails.patch], against
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
> fef9cde752a.
> {{OzoneManagerStateMachine.notifyLogFailed}} now calls
> {{ExitUtils.terminate(1, ...)}} with the failed entry (or the segment) and
> the cause, as SCM does and as OM already does for fatal apply errors. Status
> 1 matches what HDDS-17102 asks for abnormal OM stops. OM log entries carry no
> state machine data, so the other caller of this callback (reading state
> machine data) does not apply to OM.
> Covered by {{testNotifyLogFailedTerminates}} in the existing
> {{TestOzoneManagerStateMachine}}: it asserts an exit with status 1 and the
> given cause for a failed segment, and an exit with status 1 naming the term
> and index for a failed entry. Without the change it fails because nothing is
> thrown. With it the suite (65 tests) and {{TestOzoneManagerRatisServer}}
> pass, checkstyle is clean. With the patch the reproduction fails at its first
> assertion, because the failed OM requests exit status 1 as soon as the
> segment roll fails. A real JVM exit was not exercised; the log worker's close
> waits at most 3 seconds, so the shutdown hooks are not expected to hang (SCM
> uses the same pattern).
> Found by code review of the OM and SCM Ratis state machine integration
> (leader readiness, snapshot install and log failure handling), as part of the
> TLA+ verification effort under HDDS-15926, on commit
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
