Siyao Meng created HDDS-17047:
---------------------------------
Summary: Failed rename while Recon swaps in its rebuilt DB leaves
no open DB; Recon DB reads and later rebuilds fail until restart
Key: HDDS-17047
URL: https://issues.apache.org/jira/browse/HDDS-17047
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-9-restore-current-db-on-failed-swap.patch,
TestBugCR9FailedDbSwap.java
h3. Mechanism
{{ReconDBProvider.replaceStagedDb}} deletes {{.backup}}, closes the live DB,
renames the live DB to {{.backup}} and the staged DB to the live path, then
opens it. Its rollback, run when {{dbStore}} is null, always renames the live
path to {{.staged}} and {{.backup}} to the live path, whichever renames
actually happened:
* If the first rename fails, the live DB is still in place but closed; the
rollback's rename of the live DB to {{.staged}} fails as well (the staged DB is
still there).
* If the second rename fails, the live path does not exist; the rollback fails
on it and {{.backup}} is not restored.
The rollback's exception replaces the original one and {{dbStore}} stays null.
Separately, {{initializeDBStore}} returns null without an exception when the
swapped DB cannot be opened, so the swap can also return normally with
{{dbStore}} null.
{{ReconTaskControllerImpl.reInitializeTasks}} then calls
{{ReconNamespaceSummaryManagerImpl.reinitialize}}, which throws a
NullPointerException outside its {{catch (IOException)}}, before the container
metadata, global stats and file metadata managers are reinitialized. All four
managers keep the closed DB, and every later swap throws a NullPointerException
at {{dbStore.getDbLocation()}}.
h3. Trigger
A rename in the Recon DB directory fails during a swap after the {{.backup}}
delete succeeded. Because {{.backup}} stays in place after each successful
swap, a directory wide fault (permission change, read only remount) usually
fails that delete first, before anything is closed, which is harmless. The bug
is therefore reached on the first swap of an installation, by a fault that hits
the rename but not the delete, or by a RocksDB open failure of the swapped DB.
h3. Impact
Until Recon restarts: reads of the Recon DB (namespace summaries, container key
mappings, OM insight counters, file size counts) fail with "Rocks Database is
closed", and every later rebuild fails at the swap. A restart restores
{{.backup}} when it exists. No data is lost.
Related but different: HDDS-17037 covers reads that see the closed DB during a
successful swap.
h3. Reproduction
PASS on unmodified source, deterministic. [^TestBugCR9FailedDbSwap.java]
({{hadoop-ozone/recon}}) uses a real {{ReconDBProvider}}:
* {{renameOfStagedDbFails}}: the staged directory is removed before the swap,
so the second rename fails. Afterwards no DB is open, the live path does not
exist, {{.backup}} is left, {{reinitialize}} throws NullPointerException, a
read fails with a closed DB, and the next swap throws NullPointerException. A
new provider (restart) restores {{.backup}}.
* {{renameOfCurrentDbFails}}: on the first swap the Recon DB directory is made
unwritable, so the first rename fails. Afterwards no DB is open although the
live DB is intact, and the next swap throws NullPointerException. Skipped where
the directory cannot be made unwritable (for example as root).
A passing test means the defect is present.
h3. Patch
[^CR-9-restore-current-db-on-failed-swap.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
A null result of opening the swapped DB now throws, and the rollback undoes
only what happened: {{.backup}} exists only if the live DB was moved, so then
the live path (if present) goes back to {{.staged}} and {{.backup}} back to the
live path; then the live path is reopened. A failed rename inside the rollback
is added to the original exception as suppressed, and the DB is then not
reopened, so that an absent live path is not recreated empty and the restart
can still restore {{.backup}}. A failed reopen is added as suppressed too. A
successful swap is unchanged.
{{ReconTaskControllerImpl.reInitializeTasks}} now returns false when the swap
throws, so the controller retries the rebuild through its existing failure path
(a new checkpoint after the retry delay, a full OM snapshot after repeated
failures). Before, it reported success and kept the old derived data, which
then permanently missed the events drained when the rebuild was queued.
Behaviour change: a failing {{.backup}} delete, which already left the old DB
in service, now also triggers these retries instead of being ignored.
Covered by {{TestReconDBProvider#testFailedSwapKeepsCurrentDb}} (second rename
fails), {{TestReconDBProvider#testUnopenableSwappedDbKeepsCurrentDb}} (the
swapped DB has a corrupt {{CURRENT}} file) and
{{TestReconTaskControllerImpl#testReInitializeTasksFailsWhenDbSwapFails}},
which all fail without the change. With it, {{TestReconDBProvider}},
{{TestReconReplicaHistoryDBProvider}},
{{TestOmTableInsightTaskStaleCounterAfterReinit}},
{{TestReconTaskControllerImpl}}, {{TestContainerEndpoint}},
{{TestReconVersionManager}} and {{TestNSSummaryUnifiedControl}} pass (93 tests,
1 skipped as before) and checkstyle is clean. With the patch both reproduction
tests fail because the previous DB is open again after the failed swap.
Found by code review of the Recon namespace summary and task sync paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
