Sadanand Shenoy created HDDS-16069:
--------------------------------------
Summary: OM bootstrap v2 checkpoint hardLinkFile uses wrong paths
when leader ozone.metadata.dirs is wrong
Key: HDDS-16069
URL: https://issues.apache.org/jira/browse/HDDS-16069
Project: Apache Ozone
Issue Type: Bug
Reporter: Sadanand Shenoy
Assignee: Sadanand Shenoy
h3. Problem
During OM HA bootstrap, the bootstrapping node can fail to install the leader
checkpoint with:
FileAlreadyExistsException: .../om.db.candidate/../omNode-1/om.db/000014.sst ->
.../(dev=...,ino=...)-...
{{hardLinkFile}} contained paths like {{../omNode-1/om.db/000034.sst}} instead
of the expected {{om.db/000034.sst}} and {{{}db.snapshots/...{}}}.
Follower {{installHardLinks}} resolves column 1 from {{{}om.db.candidate{}}}.
Paths with {{../omNode-1/om.db/}} normalize to
{{om.ratis.snapshot/omNode-1/om.db/}} — outside {{{}om.db.candidate{}}}.
Cleanup only deletes the candidate dir, so partial installs survive and retries
hit {{{}FileAlreadyExistsException{}}}.
h3. Root cause
{{writeHardlinkFile}} anchors paths with {{{}OMStorage.getOmDbDir(conf){}}},
which reads config, not the live DB location.
When the leader’s {{ozone.metadata.dirs}} points at the wrong directory,
{{relativize}} produces {{{}../omNode-1/om.db/...{}}}:
||Anchor ({{{}getOmDbDir{}}})||File on disk||{{hardLinkFile}} column 1||
|{{.../omNode-1}}
(correct)|{{.../omNode-1/om.db/000034.sst}}|{{om.db/000034.sst}}|
|{{.../ozone-metadata}}
(wrong)|{{.../omNode-1/om.db/000034.sst}}|{{../omNode-1/om.db/000034.sst}}|
Observed trigger in tests: {{MiniOzoneHAClusterImpl.updateOMConfigs()}}
replaced each active OM’s full config with the shared cluster template during
bootstrap. That overwrote per-node {{ozone.metadata.dirs}}
({{{}.../omNode-1{}}}) with cluster-level {{.../ozone-metadata}} while RocksDB
files remained under {{{}omNode-1/{}}}.
h3. Proposed fix
# Leader: Anchor {{writeHardlinkFile}} on
{{getDbStore().getDbLocation().getParent()}} (OM metadata top level), not
{{{}getOmDbDir(conf){}}}, so {{hardLinkFile}} stays correct regardless of
config reload.
# Follower (defense in depth): If a {{hardLinkFile}} path escapes
{{om.db.candidate}} (legacy/mis-anchored {{../...}} entries), re-resolve under
candidate; make link creation idempotent for Ratis retries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
