[
https://issues.apache.org/jira/browse/HDDS-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16916:
------------------------------
Attachment: TestBugCR1LeaderReadyBlockingWrite.java
> New SCM leader blocks its Ratis log applying thread and terminates when the
> root CA post-processing state is stale
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16916
> URL: https://issues.apache.org/jira/browse/HDDS-16916
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-1-delete-post-rotation-state-off-the-notifying-thread.patch,
> TestBugCR1LeaderReadyBlockingWrite.java
>
>
> h3. Mechanism
> {{SCMStateMachine.notifyLeaderReady}} runs on the Ratis
> {{StateMachineUpdater}} thread and notifies the SCM services inline.
> {{RootCARotationManager.notifyStatusChanged}} then calls
> {{checkAndHandlePostProcessing}} on that thread. After a root CA rotation the
> leader saves the new root certificate as the stateful configuration of
> {{RootCARotationManager}} and schedules its deletion one
> {{hdds.x509.rootca.certificate.polling.interval}} (default 2h) later. If a
> new leader still finds the entry after that window, which it measures from
> the {{notBefore}} of the root certificate, {{checkAndHandlePostProcessing}}
> calls {{deleteConfiguration}}. That is a replicated call which waits until
> its log entry is applied, and the only thread that can apply it is the one
> waiting. After {{ozone.scm.ha.ratis.request.timeout}} (default 30s) the call
> fails, {{notifyStatusChanged}} rethrows the failure as a
> {{RuntimeException}}, Ratis closes the server because the
> {{StateMachineUpdater}} caught a {{Throwable}}, and {{SCMStateMachine.close}}
> terminates the SCM.
> The same inline delete exists in the branch for an entry whose root
> certificate is older than the local one (code reading, exercised at unit
> level by the test in the patch).
> The same inline call is in the release tags ozone-1.4.0, 1.4.1, 2.0.0, 2.1.0,
> 2.1.1, 2.1.2, 2.2.0 and 2.2.1 (read, not run). In 1.4.0 and 1.4.1 the
> services are notified from {{SCMStateMachine.notifyTermIndexUpdated}}, which
> the stack trace in HDDS-11750 shows on the same thread.
> h3. Trigger
> # Secure SCM HA with {{hdds.x509.ca.rotation.enabled=true}} (default false).
> # A root CA rotation completes.
> # The leader that ran the rotation loses leadership before its scheduled
> deletion runs, which is one polling interval after the rotation finished.
> Run: the leader stops one second before the end of the window. By code
> reading a step down has the same effect, because it cancels the deletion task.
> # The next SCM becomes leader ready, out of safe mode, later than one polling
> interval after the {{notBefore}} of the new root certificate. A leader change
> that is complete before that point is harmless, because the new leader then
> only schedules the deletion for the rest of the window.
> The points in time of steps 3 and 4 differ. The new root certificate is
> created when the rotation starts and the leader schedules the deletion when
> the rotation finishes, so the point of step 4 comes first by the time the
> rotation took. That was about 1.3s in the run. By code reading it is limited
> by the wait for the sub CA acknowledgements,
> {{hdds.x509.ca.rotation.ack.timeout}} (default 15 minutes), plus the
> replicated calls of the rotation, and a leader election between the two
> points is enough, with no time without a leader. An admin leadership transfer
> does not get there, because the leader refuses it while its deletion is
> pending (run inside the window).
> If the scheduled deletion itself fails, the failure is only logged and not
> retried (code reading). The entry then stays, and the next SCM that becomes
> leader ready out of safe mode is affected whenever that happens.
> By code reading, an SCM that becomes leader while still in safe mode is not
> affected, because the check then runs later on the thread that leaves safe
> mode.
> h3. Impact
> * The new leader applies no log entry for 30s and then stops. In the run the
> SCM was stopped from its {{StateMachineUpdater}} thread and reached
> {{ExitUtils.terminate}} with status 0 ("scm statemachine is closed by ratis,
> terminate SCM"). The run used the mini cluster, where the JVM exit is
> disabled. A separate SCM process was not run.
> * Nothing brings the SCM back, it stays down until it is restarted.
> * In the run the previous leader was down as well, so the three SCM ring was
> left with one SCM and no quorum. If the previous leader is still up, two SCMs
> remain and can elect a leader (argued, not run).
> * No replicated state is damaged. The delete entry stays in the Ratis log, so
> it should be applied under the next leader and the failure should not repeat
> (argued, not run).
> * Rotation happens about once per root certificate lifetime
> ({{hdds.x509.max.duration}}, default P1865D). The leader change has to
> straddle the end of the window after it (the old leader loses leadership
> before its deletion runs, the next leader becomes ready after the window), or
> come at any later time if the scheduled deletion failed.
> h3. Reproduction
> PASS, unmodified source, nothing injected.
> [^TestBugCR1LeaderReadyBlockingWrite.java]
> ({{hadoop-ozone/integration-test}}) starts a secure mini cluster with three
> SCMs. Certificate lifetimes are shortened so that the leader rotates the root
> CA two minutes after SCM init, and the polling interval is 20s instead of 2h.
> The test stops the leader one second before the window ends and then finds
> the {{StateMachineUpdater}} thread of the new leader inside
> {{deleteConfiguration}}, called from {{SCMStateMachine.notifyLeaderReady}}.
> 30s later the Ratis division of the new leader is closing and the SCM stops.
> The test takes about three minutes. A passing test means the defect is
> present.
> h3. Patch
> [^CR-1-delete-post-rotation-state-off-the-notifying-thread.patch], against
> b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
> 5108acff480. The patch is written against that commit alone, so other fixes
> proposed for {{RootCARotationManager}} and its unit test will need a manual
> merge with it.
> {{checkAndHandlePostProcessing}} hands its two replicated deletes (the entry
> past its window and the entry older than the local root certificate) to the
> executor of the manager, which already runs the scheduled deletion. A failed
> delete is logged like a failed scheduled deletion, and the entry is checked
> again after the next leader change. Everything else stays on the notifying
> thread and is unchanged: reading the entry, setting the flag that blocks
> certificate signing inside the window, and any failure of the read or of the
> comparison, for example an entry naming a newer root certificate than the
> local one, which ends the SCM as before.
> {{SecretKeyManagerService.notifyStatusChanged}} in the same package also runs
> its replicated call on its own executor.
> Covered in the existing {{TestRootCARotationManager}} by
> {{testStalePostProcessingStateIsDeletedOffNotifyingThread}}, which fails for
> both deletes without the change because the delete runs on the notifying
> thread and its failure is thrown to the caller, and by
> {{testFailedPostProcessingCheckIsThrownToNotifier}}, which holds the
> unchanged failure of the check in place and passes with and without the
> change. With the change the SCM HA and root CA rotation unit suites of
> {{hadoop-hdds/server-scm}} pass (115 tests) and checkstyle is clean. In the
> reproduction the new leader is then ready 5.6s after the old leader stopped,
> deletes the entry from the executor thread 6ms after the check and keeps
> running, so the reproduction fails at its first assertion.
> Related issues, none of which covers this call. HDDS-11750 reported the same
> self wait for {{LegacyReplicationManager.notifyStatusChanged}} and was
> resolved as Won't Fix without a code change, and that class has since been
> removed. HDDS-16029 (open) is a different blocking call,
> {{ContainerBalancer}} joining its thread when the SCM loses leadership.
> HDDS-16222 (open) proposes a static check that {{notifyStatusChanged}} cannot
> block and names those two issues, not this call.
> Found by code review of the SCM root CA rotation paths, as part of the TLA+
> verification effort under HDDS-15926, on commit
> b9b33851b20fa11baf4b07982de9afb8ea44f85a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
