Siyao Meng created HDDS-16878:
---------------------------------
Summary: OM checkpoint servlets keep using the DB store closed by
a checkpoint install and return HTTP 500 once that OM is leader
Key: HDDS-16878
URL: https://issues.apache.org/jira/browse/HDDS-16878
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-checkpoint-servlets-read-current-db-store.patch,
TestBugCR1CheckpointServletClosedStore.java
h3. Mechanism
{{DBCheckpointServlet.initialize}} stores the {{DBStore}} it is given, and
{{OMDBCheckpointServlet.init}} and {{OMDBCheckpointServletInodeBasedXfer.init}}
pass {{om.getMetadataManager().getStore()}}. Both servlets then take every
checkpoint, the DB location and the {{RocksDBCheckpointDiffer}} from
{{getDbStore()}}, which returns that stored field.
{{OzoneManager.installCheckpoint}} stops the old metadata manager, which closes
its store, and {{reloadOMState}} creates a new {{OmMetadataManagerImpl}}. The
HTTP server and its servlets are created once in {{OzoneManager.start}} and are
never initialized again, so after the install the servlets still use the closed
store.
Jetty initializes each servlet separately, on its own first request. Each
endpoint is affected if it served at least one request before the install in
the same process lifetime: for example while that OM was leader earlier (a
follower install, an OM bootstrap or a Recon full snapshot), or a request that
reached it as a follower and got 503. Followers and bootstrapping OMs download
from {{/v2/dbCheckpoint}}.
The same code is in every release since 2.0.0 (code reading of the release
tags).
h3. Trigger
# OM X serves a request on a checkpoint endpoint (as leader, or 503 as
follower). Only the endpoints that served a request are affected.
# X falls behind far enough that the leader sends a snapshot, and X installs
the checkpoint through {{installSnapshotFromLeader}}.
# X becomes leader (election or leadership transfer).
h3. Impact
* While X is leader, every request to {{/dbCheckpoint}} and
{{/v2/dbCheckpoint}} returns HTTP 500 and the OM logs {{RocksDatabaseException:
Rocks Database is closed}} from {{RDBStore.getCheckpoint}}. This lasts until X
restarts.
* A follower that needs a checkpoint from X cannot get one:
{{OmRatisSnapshotProvider.downloadDBSnapshotFromLeader}} fails with "ErrorCode:
500" (run). A bootstrapping OM and a Recon full snapshot use the same endpoints
and would fail the same way (code reading, not run). Recon incremental sync
through {{getDBUpdates}} reads the current store and is not affected.
* No client data is affected.
h3. Reproduction
PASS on the unmodified source, deterministic.
[^TestBugCR1CheckpointServletClosedStore.java]
({{hadoop-ozone/integration-test}}) starts three OMs with one held back,
advances the leader past the Ratis log purge gap, and starts the late OM with
the existing {{OzoneManager.setTestInstallSnapshot}} hook set only until both
servlets have answered one request (503). Ratis then installs the checkpoint
through the real path, leadership is transferred to that OM, and both endpoints
plus a real follower download are tried. Output:
{noformat}
Before install, omNode-3 answered /dbCheckpoint -> 503, /v2/dbCheckpoint ->
503. After install: store replaced = true, old store closed = true
After omNode-3 became leader: /dbCheckpoint -> 500, /v2/dbCheckpoint -> 500,
follower download failed: Unexpected exception when trying to reach OM to
download latest checkpoint. Checkpoint URL:
http://127.0.0.1:15017/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true.
ErrorCode: 500
{noformat}
A passing test means the defect is present. With the patch both endpoints
return 200, the download succeeds and the test fails at its first defect
assertion.
h3. Patch
[^CR-1-checkpoint-servlets-read-current-db-store.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
6818707698c.
Both OM servlets override {{getDbStore()}} to return the store of the OM's
current metadata manager, the same way their bootstrap lock and
{{getSnapshotDir()}} already read the OM's current state. Nothing changes for
an OM that never installed a checkpoint. {{DBCheckpointServlet}} and the SCM
servlet are not changed. Four places in the existing servlet tests injected a
spy store through {{initialize}}; they now also stub {{getDbStore()}} with it.
Covered by the new
{{TestOMRatisSnapshots#testDbCheckpointServletsServeInstalledDbAfterLeaderChange}},
which fails without the change with {{[/dbCheckpoint] expected: 200 but was:
500}}. With the change {{TestOMRatisSnapshots}}, {{TestOMDbCheckpointServlet}},
{{TestOMDbCheckpointServletInodeBasedXfer}}, {{TestGetDBUpdatesAuthorization}}
and {{TestOMDBCheckpointServletInodeBasedXferNonLeader}} pass (37 tests) and
checkstyle is clean.
{{SCMDBCheckpointServlet}} caches the SCM store the same way, and an SCM
checkpoint install also creates a new store (code reading, not run). Recon
fetches the SCM DB over gRPC, so only direct use of the SCM endpoint is
affected; it is left for a separate change. Other holders of the old metadata
manager, such as the snapshot directory metrics and the multi tenant Ranger
sync service (HDDS-16853), are out of scope here.
Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
