[
https://issues.apache.org/jira/browse/HDDS-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17047:
------------------------------
Attachment: TestBugCR9FailedDbSwap.java
> Failed rename while Recon swaps in its rebuilt DB leaves no open DB; Recon DB
> reads and later rebuilds fail until restart
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17047
> URL: https://issues.apache.org/jira/browse/HDDS-17047
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-9-restore-current-db-on-failed-swap.patch,
> TestBugCR9FailedDbSwap.java
>
>
> h3. Mechanism
> {{ReconDBProvider.replaceStagedDb}} deletes {{.backup}}, closes the live DB,
> renames the live DB to {{.backup}} and the staged DB to the live path, then
> opens it. Its rollback, run when {{dbStore}} is null, always renames the live
> path to {{.staged}} and {{.backup}} to the live path, whichever renames
> actually happened:
> * If the first rename fails, the live DB is still in place but closed; the
> rollback's rename of the live DB to {{.staged}} fails as well (the staged DB
> is still there).
> * If the second rename fails, the live path does not exist; the rollback
> fails on it and {{.backup}} is not restored.
> The rollback's exception replaces the original one and {{dbStore}} stays
> null. Separately, {{initializeDBStore}} returns null without an exception
> when the swapped DB cannot be opened, so the swap can also return normally
> with {{dbStore}} null.
> {{ReconTaskControllerImpl.reInitializeTasks}} then calls
> {{ReconNamespaceSummaryManagerImpl.reinitialize}}, which throws a
> NullPointerException outside its {{catch (IOException)}}, before the
> container metadata, global stats and file metadata managers are
> reinitialized. All four managers keep the closed DB, and every later swap
> throws a NullPointerException at {{dbStore.getDbLocation()}}.
> h3. Trigger
> A rename in the Recon DB directory fails during a swap after the {{.backup}}
> delete succeeded. Because {{.backup}} stays in place after each successful
> swap, a directory wide fault (permission change, read only remount) usually
> fails that delete first, before anything is closed, which is harmless. The
> bug is therefore reached on the first swap of an installation, by a fault
> that hits the rename but not the delete, or by a RocksDB open failure of the
> swapped DB.
> h3. Impact
> Until Recon restarts: reads of the Recon DB (namespace summaries, container
> key mappings, OM insight counters, file size counts) fail with "Rocks
> Database is closed", and every later rebuild fails at the swap. A restart
> restores {{.backup}} when it exists. No data is lost.
> Related but different: HDDS-17037 covers reads that see the closed DB during
> a successful swap.
> h3. Reproduction
> PASS on unmodified source, deterministic. [^TestBugCR9FailedDbSwap.java]
> ({{hadoop-ozone/recon}}) uses a real {{ReconDBProvider}}:
> * {{renameOfStagedDbFails}}: the staged directory is removed before the swap,
> so the second rename fails. Afterwards no DB is open, the live path does not
> exist, {{.backup}} is left, {{reinitialize}} throws NullPointerException, a
> read fails with a closed DB, and the next swap throws NullPointerException. A
> new provider (restart) restores {{.backup}}.
> * {{renameOfCurrentDbFails}}: on the first swap the Recon DB directory is
> made unwritable, so the first rename fails. Afterwards no DB is open although
> the live DB is intact, and the next swap throws NullPointerException. Skipped
> where the directory cannot be made unwritable (for example as root).
> A passing test means the defect is present.
> h3. Patch
> [^CR-9-restore-current-db-on-failed-swap.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> A null result of opening the swapped DB now throws, and the rollback undoes
> only what happened: {{.backup}} exists only if the live DB was moved, so then
> the live path (if present) goes back to {{.staged}} and {{.backup}} back to
> the live path; then the live path is reopened. A failed rename inside the
> rollback is added to the original exception as suppressed, and the DB is then
> not reopened, so that an absent live path is not recreated empty and the
> restart can still restore {{.backup}}. A failed reopen is added as suppressed
> too. A successful swap is unchanged.
> {{ReconTaskControllerImpl.reInitializeTasks}} now returns false when the swap
> throws, so the controller retries the rebuild through its existing failure
> path (a new checkpoint after the retry delay, a full OM snapshot after
> repeated failures). Before, it reported success and kept the old derived
> data, which then permanently missed the events drained when the rebuild was
> queued. Behaviour change: a failing {{.backup}} delete, which already left
> the old DB in service, now also triggers these retries instead of being
> ignored.
> Covered by {{TestReconDBProvider#testFailedSwapKeepsCurrentDb}} (second
> rename fails), {{TestReconDBProvider#testUnopenableSwappedDbKeepsCurrentDb}}
> (the swapped DB has a corrupt {{CURRENT}} file) and
> {{TestReconTaskControllerImpl#testReInitializeTasksFailsWhenDbSwapFails}},
> which all fail without the change. With it, {{TestReconDBProvider}},
> {{TestReconReplicaHistoryDBProvider}},
> {{TestOmTableInsightTaskStaleCounterAfterReinit}},
> {{TestReconTaskControllerImpl}}, {{TestContainerEndpoint}},
> {{TestReconVersionManager}} and {{TestNSSummaryUnifiedControl}} pass (93
> tests, 1 skipped as before) and checkstyle is clean. With the patch both
> reproduction tests fail because the previous DB is open again after the
> failed swap.
> Found by code review of the Recon namespace summary and task sync paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
