Siyao Meng created HDDS-16961:
---------------------------------
Summary: ozone admin om fetch-key does not make OM fetch the
current secret key from SCM, it prints the key OM already has
Key: HDDS-16961
URL: https://issues.apache.org/jira/browse/HDDS-16961
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch,
TestBugCR11FetchKeyDoesNotReachSigner.java
h3. Mechanism
{{ozone admin om fetch-key}} is described as "CLI command to force OM to fetch
the latest secret key from SCM". {{OzoneManager.refetchSecretKey}} serves it
with two calls on its secret key client: {{refetchSecretKey()}}, then
{{getCurrentSecretKey().getId()}}, which the command prints as "Current Secret
Key ID".
Since HDDS-8829 that client is a {{DefaultSecretKeyClient}}, the composition of
a signer client and a verifier client. It passes {{getCurrentSecretKey}},
{{start}} and {{stop}} on to {{DefaultSecretKeySignerClient}}, but not
{{refetchSecretKey}}. So the empty default of
{{SecretKeySignerClient.refetchSecretKey}} runs, SCM is not asked, and the
command reports the ID of the key OM already holds, with exit code 0.
When HDDS-8677 added the command, OM held a {{DefaultSecretKeySignerClient}}
directly, whose {{refetchSecretKey}} does ask SCM. The command worked in 1.4.x
and has had no effect since 2.0.0.
h3. Trigger
# Secure cluster.
# SCM gets a new current secret key that OM does not have yet, for example
through {{ozone admin scm rotate --force}}.
# {{ozone admin om fetch-key}} is run.
h3. Impact
* The command reports success and a key ID, but OM keeps signing with its old
key until its own poller fetches the new one, which happens once the old key is
older than {{hdds.secret.key.rotate.duration}} (1d). OM cannot be made to
switch earlier, which is what the command is for, short of restarting it.
* The printed ID is not SCM's current key, without any hint.
* Nothing else is affected. The old key stays valid on SCM until it expires, so
tokens that OM issues in the meantime are accepted.
h3. Reproduction
PASS, deterministic, unmodified source, under 1 second.
[^TestBugCR11FetchKeyDoesNotReachSigner.java] ({{hadoop-hdds/framework}})
creates the client the way OM does, with a mock SCM that has a new current key
after the first fetch, and makes the two calls of
{{OzoneManager.refetchSecretKey}}. The key is unchanged and SCM was called once
(by {{start}}). A second test is the control: the same sequence on
{{DefaultSecretKeySignerClient}} alone returns the new key. A passing test
means the defect is present.
On a secure mini cluster the command itself was run through
{{TestBlockTokensCLI.testFetchKeyOMAdminCommand}} with a forced rotation on SCM
added before it (this is the regression test of the patch): without the fix it
fails with "expected: <3f7f03d9-...> but was: <e13b7dfc-...>", the first being
SCM's current key and the second the ID that the command printed.
h3. Patch
[^CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch], against
025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at
4389803780f.
{{DefaultSecretKeyClient.refetchSecretKey}} now passes the call on to the
signer client, as it does for the other signer methods. The verifier side is
not touched. {{OzoneManager.refetchSecretKey}} rethrows the cause of the
{{UncheckedIOException}} of the signer client, so that a failed fetch from SCM
is returned to the caller as an error.
{{TestBlockTokensCLI.testFetchKeyOMAdminCommand}} now rotates the key on SCM
first, so that the printed ID can only match if OM really fetched it. The
existing test compared the printed ID with SCM's current key without a rotation
in between, which is why it kept passing. {{TestOMAdminRpcAuthorization}} gets
a test in which the fetch fails and the caller has to receive the error.
With the patch {{TestBlockTokensCLI}} (3 tests),
{{TestOMAdminRpcAuthorization}} (5 tests) and {{TestBlockTokens}} (5 tests)
pass on a mini cluster, the secret key and token unit suites of
{{hadoop-hdds/framework}} (38 tests) pass, and checkstyle is clean. Without the
two main changes the changed test and the new test fail. With the patch the
reproduction fails at its assertion that the key is unchanged.
Found by code review of the SCM secret key rotation and distribution paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
