Siyao Meng created HDDS-17447:
---------------------------------
Summary: SCM terminates when it applies a deleted block retry
count Raft log entry written by SCM 2.0 or older, and again on every restart
Key: HDDS-17447
URL: https://issues.apache.org/jira/browse/HDDS-17447
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-11-apply-legacy-deleted-block-retry-count-log-entries.patch,
TestBugCR11LegacyRetryCountEntryReplay.java
h3. Mechanism
SCM 1.3.0 to 2.0.x write {{BLOCK}} Raft log entries for
{{DeletedBlockLogStateManager.increaseRetryCountOfTransactionInDB}} (when a
delete transaction goes over {{ozone.scm.block.deletion.max.retry}}, default
4096) and {{resetRetryCountOfTransactionInDB}} (on {{ozone admin scm
deletedBlocksTxn reset}}). 2.1.0 stopped writing them (HDDS-13520) and kept
both methods as no-ops so that it can still apply them. HDDS-14915 kept them
for that compatibility and HDDS-14916 (2.2.0) made them {{default}} methods of
the interface, but the generated
{{DeletedBlockLogStateManagerInvoker.invokeLocal}} has no case for default
methods. Applying either entry throws {{IllegalArgumentException("Method not
found: increaseRetryCountOfTransactionInDB in DeletedBlockLogStateManager")}}.
That is not an {{SCMException}}, so {{SCMStateMachine.applyTransaction}} calls
{{ExitUtils.terminate}}. The entry is still unapplied, so the SCM replays it
and terminates again on every start. The request encoding is unchanged since
2.0.0 (checked byte for byte in the reproduction), only the method name lookup
fails. HDDS-17421 is the same failure for the pre ZDU finalization entries, and
HDDS-16169 fixed it for the renamed
{{ContainerStateManager.updateContainerState}}.
h3. Trigger
An SCM on 2.2.x or master has to apply such an entry, which happens when it did
not apply it before the upgrade:
# An SCM that was down or behind when its ring wrote the entry on 1.3.x to
2.0.x, and is then started on 2.2.x or later. It catches up by log append.
# An SCM stopped uncleanly (crash, kill) after the entry and before a
transaction buffer flush or snapshot covered it, then started on 2.2.x or
later. A graceful stop takes a snapshot first and is not affected.
An upgrade through 2.1.x is not affected, because 2.1.x applies both entries.
h3. Impact
The affected SCM cannot start on 2.2.0, 2.2.1 or master. If all SCMs were
stopped uncleanly right after such an entry, no SCM can start. No data loss. A
workaround (argued, not run) is to start the stuck SCM on its old release or on
2.1.x until it has applied the entry, then upgrade it again.
h3. Reproduction
PASS on unmodified source. [^TestBugCR11LegacyRetryCountEntryReplay.java]
({{hadoop-hdds/server-scm}}) starts one real SCM with its own single node Ratis
ring, checks that the entry it builds is byte for byte the one SCM 2.0.0
writes, appends it through the SCM's Ratis server, and restarts the SCM once.
For both methods, the SCM terminates with {{Method not found: <method> in
DeletedBlockLogStateManager}} on apply and again after the restart. A passing
test means the defect is present. Not run against a real 2.0.x ring.
h3. Patch
[^CR-11-apply-legacy-deleted-block-retry-count-log-entries.patch], against
master d5397d675e6. It also applies to ozone-2.2 at 7dbea64856d, and together
with the HDDS-17421 patch.
As in HDDS-16169, both methods are abstract again in
{{DeletedBlockLogStateManager}}, still {{@Deprecated}} and {{@Replicate}},
{{DeletedBlockLogStateManagerImpl}} implements them as no-ops, and
{{DeletedBlockLogStateManagerInvoker}} is regenerated.
{{resetRetryCountOfTransactionInDB}} now returns {{void}}: with a case for it
but the old {{int}} return type, applying the entry still terminates the SCM,
with {{Failed to resolve int}}, because {{SCMRatisResponse.encode}} cannot
encode a primitive return type. The old return value has no caller. No Raft log
or proto change.
{{TestScmInvokerCodeGenerator}} now also checks, for every SCM handler
interface, that each {{@Replicate}} method has a case in the generated
{{invokeLocal}} and does not return a primitive. Without the change it fails
for {{DeletedBlockLogStateManager}}, and it would catch the next method made
{{default}} the same way. With it, {{TestScmInvokerCodeGenerator}},
{{TestDeletedBlockLog}}, {{TestSCMStateMachine}} and
{{TestSCMHATransactionBufferMonitorTask}} pass, and checkstyle is clean. With
the patch the reproduction fails at its first assertion, because both entries
are applied.
Found while validating the results of HDDS-17231, part of the TLA+ verification
effort under HDDS-15926, on commit d5397d675e6. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Claude Code (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
