[
https://issues.apache.org/jira/browse/HDDS-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh resolved HDDS-15995.
---------------------------------------
Release Note: This is invalid and not an issue.
Resolution: Not A Problem
> Recon SCM RocksDB closes with background IO error because snapshot DB
> directory is renamed while the DB is still open
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15995
> URL: https://issues.apache.org/jira/browse/HDDS-15995
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.1.1
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> *Summary*
> During Recon's SCM DB snapshot swap, the downloaded snapshot directory
> (scm.snapshot.db_<timestamp>) is renamed to the permanent name (recon-scm.db)
> *while the RocksDB instance is still open on it*. RocksDB keeps its original
> directory path internally, so a later attempt to create a new WAL/log file
> fails with "No such file or directory", RocksDB sets a background IO error and
> shuts itself down. All subsequent Recon SCM background work (PipelineSyncTask,
> container tasks) then fails with "Rocks Database is closed".
> *Observed RocksDB LOG (from Recon SCM DB folder)*
> {code:java}
> New memtable created with log file: #1517. Immutable memtables: 0.
> [WARN] Background IO error IO error: No such file or directory: While open a
> file
> for appending: <reconScmDbDir>/scm.snapshot.db_1782737001797/001517.log: No
> such file or directory
> ErrorHandler: Set regular background error
> Shutdown: canceling all background work
> Shutdown complete
> {code}
> *How to reproduce*
> # Enable snapshot sync and set ozone.recon.scm.container.threshold=0.
> # Set a small memtable size
> (ozone.metastore.rocksdb.cf.write.buffer.size=64KB)
> to make WAL rotation happen quickly.
> # Create at least one container in SCM (write a key).
> # Wipe Recon's SCM DB (recon-scm.db) and restart Recon so it downloads a fresh
> snapshot -> triggers the rename-while-open.
> # Generate continued write activity (create keys) and observe the Recon SCM
> RocksDB LOG showing the background IO error on the
> scm.snapshot.db_<timestamp>
> path, followed by "Rocks Database is closed" in Recon logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
