Siyao Meng created HDDS-16916:
---------------------------------
Summary: New SCM leader blocks its Ratis log applying thread and
terminates when the root CA post-processing state is stale
Key: HDDS-16916
URL: https://issues.apache.org/jira/browse/HDDS-16916
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-1-delete-post-rotation-state-off-the-notifying-thread.patch,
TestBugCR1LeaderReadyBlockingWrite.java
h3. Mechanism
{{SCMStateMachine.notifyLeaderReady}} runs on the Ratis {{StateMachineUpdater}}
thread and notifies the SCM services inline.
{{RootCARotationManager.notifyStatusChanged}} then calls
{{checkAndHandlePostProcessing}} on that thread. After a root CA rotation the
leader saves the new root certificate as the stateful configuration of
{{RootCARotationManager}} and schedules its deletion one
{{hdds.x509.rootca.certificate.polling.interval}} (default 2h) later. If a new
leader still finds the entry after that window, which it measures from the
{{notBefore}} of the root certificate, {{checkAndHandlePostProcessing}} calls
{{deleteConfiguration}}. That is a replicated call which waits until its log
entry is applied, and the only thread that can apply it is the one waiting.
After {{ozone.scm.ha.ratis.request.timeout}} (default 30s) the call fails,
{{notifyStatusChanged}} rethrows the failure as a {{RuntimeException}}, Ratis
closes the server because the {{StateMachineUpdater}} caught a {{Throwable}},
and {{SCMStateMachine.close}} terminates the SCM.
The same inline delete exists in the branch for an entry whose root certificate
is older than the local one (code reading, exercised at unit level by the test
in the patch).
The same inline call is in the release tags ozone-1.4.0, 1.4.1, 2.0.0, 2.1.0,
2.1.1, 2.1.2, 2.2.0 and 2.2.1 (read, not run). In 1.4.0 and 1.4.1 the services
are notified from {{SCMStateMachine.notifyTermIndexUpdated}}, which the stack
trace in HDDS-11750 shows on the same thread.
h3. Trigger
# Secure SCM HA with {{hdds.x509.ca.rotation.enabled=true}} (default false).
# A root CA rotation completes.
# The leader that ran the rotation loses leadership before its scheduled
deletion runs, which is one polling interval after the rotation finished. Run:
the leader stops one second before the end of the window. By code reading a
step down has the same effect, because it cancels the deletion task.
# The next SCM becomes leader ready, out of safe mode, later than one polling
interval after the {{notBefore}} of the new root certificate. A leader change
that is complete before that point is harmless, because the new leader then
only schedules the deletion for the rest of the window.
The points in time of steps 3 and 4 differ. The new root certificate is created
when the rotation starts and the leader schedules the deletion when the
rotation finishes, so the point of step 4 comes first by the time the rotation
took. That was about 1.3s in the run. By code reading it is limited by the wait
for the sub CA acknowledgements, {{hdds.x509.ca.rotation.ack.timeout}} (default
15 minutes), plus the replicated calls of the rotation, and a leader election
between the two points is enough, with no time without a leader. An admin
leadership transfer does not get there, because the leader refuses it while its
deletion is pending (run inside the window).
If the scheduled deletion itself fails, the failure is only logged and not
retried (code reading). The entry then stays, and the next SCM that becomes
leader ready out of safe mode is affected whenever that happens.
By code reading, an SCM that becomes leader while still in safe mode is not
affected, because the check then runs later on the thread that leaves safe mode.
h3. Impact
* The new leader applies no log entry for 30s and then stops. In the run the
SCM was stopped from its {{StateMachineUpdater}} thread and reached
{{ExitUtils.terminate}} with status 0 ("scm statemachine is closed by ratis,
terminate SCM"). The run used the mini cluster, where the JVM exit is disabled.
A separate SCM process was not run.
* Nothing brings the SCM back, it stays down until it is restarted.
* In the run the previous leader was down as well, so the three SCM ring was
left with one SCM and no quorum. If the previous leader is still up, two SCMs
remain and can elect a leader (argued, not run).
* No replicated state is damaged. The delete entry stays in the Ratis log, so
it should be applied under the next leader and the failure should not repeat
(argued, not run).
* Rotation happens about once per root certificate lifetime
({{hdds.x509.max.duration}}, default P1865D). The leader change has to straddle
the end of the window after it (the old leader loses leadership before its
deletion runs, the next leader becomes ready after the window), or come at any
later time if the scheduled deletion failed.
h3. Reproduction
PASS, unmodified source, nothing injected.
[^TestBugCR1LeaderReadyBlockingWrite.java] ({{hadoop-ozone/integration-test}})
starts a secure mini cluster with three SCMs. Certificate lifetimes are
shortened so that the leader rotates the root CA two minutes after SCM init,
and the polling interval is 20s instead of 2h. The test stops the leader one
second before the window ends and then finds the {{StateMachineUpdater}} thread
of the new leader inside {{deleteConfiguration}}, called from
{{SCMStateMachine.notifyLeaderReady}}. 30s later the Ratis division of the new
leader is closing and the SCM stops. The test takes about three minutes. A
passing test means the defect is present.
h3. Patch
[^CR-1-delete-post-rotation-state-off-the-notifying-thread.patch], against
b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
5108acff480. The patch is written against that commit alone, so other fixes
proposed for {{RootCARotationManager}} and its unit test will need a manual
merge with it.
{{checkAndHandlePostProcessing}} hands its two replicated deletes (the entry
past its window and the entry older than the local root certificate) to the
executor of the manager, which already runs the scheduled deletion. A failed
delete is logged like a failed scheduled deletion, and the entry is checked
again after the next leader change. Everything else stays on the notifying
thread and is unchanged: reading the entry, setting the flag that blocks
certificate signing inside the window, and any failure of the read or of the
comparison, for example an entry naming a newer root certificate than the local
one, which ends the SCM as before.
{{SecretKeyManagerService.notifyStatusChanged}} in the same package also runs
its replicated call on its own executor.
Covered in the existing {{TestRootCARotationManager}} by
{{testStalePostProcessingStateIsDeletedOffNotifyingThread}}, which fails for
both deletes without the change because the delete runs on the notifying thread
and its failure is thrown to the caller, and by
{{testFailedPostProcessingCheckIsThrownToNotifier}}, which holds the unchanged
failure of the check in place and passes with and without the change. With the
change the SCM HA and root CA rotation unit suites of
{{hadoop-hdds/server-scm}} pass (115 tests) and checkstyle is clean. In the
reproduction the new leader is then ready 5.6s after the old leader stopped,
deletes the entry from the executor thread 6ms after the check and keeps
running, so the reproduction fails at its first assertion.
Related issues, none of which covers this call. HDDS-11750 reported the same
self wait for {{LegacyReplicationManager.notifyStatusChanged}} and was resolved
as Won't Fix without a code change, and that class has since been removed.
HDDS-16029 (open) is a different blocking call, {{ContainerBalancer}} joining
its thread when the SCM loses leadership. HDDS-16222 (open) proposes a static
check that {{notifyStatusChanged}} cannot block and names those two issues, not
this call.
Found by code review of the SCM root CA rotation paths, as part of the TLA+
verification effort under HDDS-15926, on commit
b9b33851b20fa11baf4b07982de9afb8ea44f85a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
