[
https://issues.apache.org/jira/browse/HDDS-15770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-15770:
------------------------------
Description:
HDDS-14225 pins the RocksDB format_version to 5
(ManagedBlockBasedTableConfig.FORMAT_VERSION), applied unconditionally in
DBProfile.SSD.getBlockBasedTableConfig() (OM/SCM/DN all route through it) and
in RDBSstFileWriter. This is deliberate: RocksDB 9+ defaults to
format_version=6 (RocksDB >= 10.4.0
[supports|https://github.com/facebook/rocksdb/blob/v10.10.1/include/rocksdb/table.h#L581-L596]
format_version=7, but the default is still 6), which cannot be read by RocksDB
< 8.6.0, so pinning to 5 keeps files readable if Ozone is downgraded before
finalization.
Once a cluster is finalized, downgrade across the layout boundary is no longer
supported, so newer files would be safe. This ticket tracks the option of
writing a newer format_version after finalization instead of staying at 5
forever.
Scope would include:
- Introduce an HDDS and OM layout feature for the RocksDB format upgrade, so
there is a real downgrade barrier protecting the newer files.
- Thread finalization/layout state into RocksDB option construction. DBProfile
is currently a static enum with no access to upgrade state, and DBs open early
in startup, so this is non-trivial plumbing rather than a flag flip.
- Choose format_version at DB open: pinned value when pre-finalized, newer
value when finalized.
Testing:
- Mixed-format DBs: files written at the old and new versions coexisting and
readable after compaction.
- Pre-finalized downgrade path still works with the pinned version.
- Non-rolling-upgrade coverage for the new layout feature.
was:
HDDS-14225 pins the RocksDB format_version to 5
(ManagedBlockBasedTableConfig.FORMAT_VERSION), applied unconditionally in
DBProfile.SSD.getBlockBasedTableConfig() (OM/SCM/DN all route through it) and
in RDBSstFileWriter. This is deliberate: RocksDB 9+ defaults to
format_version=6, which cannot be read by RocksDB < 8.6.0, so pinning to 5
keeps files readable if Ozone is downgraded before finalization.
Once a cluster is finalized, downgrade across the layout boundary is no longer
supported, so newer files would be safe. This ticket tracks the option of
writing a newer format_version after finalization instead of staying at 5
forever.
Scope would include:
- Introduce an HDDS and OM layout feature for the RocksDB format upgrade, so
there is a real downgrade barrier protecting the newer files.
- Thread finalization/layout state into RocksDB option construction. DBProfile
is currently a static enum with no access to upgrade state, and DBs open early
in startup, so this is non-trivial plumbing rather than a flag flip.
- Choose format_version at DB open: pinned value when pre-finalized, newer
value when finalized.
Testing:
- Mixed-format DBs: files written at the old and new versions coexisting and
readable after compaction.
- Pre-finalized downgrade path still works with the pinned version.
- Non-rolling-upgrade coverage for the new layout feature.
> Allow writing newer RocksDB format_version once the cluster is finalized
> ------------------------------------------------------------------------
>
> Key: HDDS-15770
> URL: https://issues.apache.org/jira/browse/HDDS-15770
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: upgrade
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
>
> HDDS-14225 pins the RocksDB format_version to 5
> (ManagedBlockBasedTableConfig.FORMAT_VERSION), applied unconditionally in
> DBProfile.SSD.getBlockBasedTableConfig() (OM/SCM/DN all route through it) and
> in RDBSstFileWriter. This is deliberate: RocksDB 9+ defaults to
> format_version=6 (RocksDB >= 10.4.0
> [supports|https://github.com/facebook/rocksdb/blob/v10.10.1/include/rocksdb/table.h#L581-L596]
> format_version=7, but the default is still 6), which cannot be read by
> RocksDB < 8.6.0, so pinning to 5 keeps files readable if Ozone is downgraded
> before finalization.
> Once a cluster is finalized, downgrade across the layout boundary is no
> longer supported, so newer files would be safe. This ticket tracks the option
> of writing a newer format_version after finalization instead of staying at 5
> forever.
> Scope would include:
> - Introduce an HDDS and OM layout feature for the RocksDB format upgrade, so
> there is a real downgrade barrier protecting the newer files.
> - Thread finalization/layout state into RocksDB option construction.
> DBProfile is currently a static enum with no access to upgrade state, and DBs
> open early in startup, so this is non-trivial plumbing rather than a flag
> flip.
> - Choose format_version at DB open: pinned value when pre-finalized, newer
> value when finalized.
> Testing:
> - Mixed-format DBs: files written at the old and new versions coexisting and
> readable after compaction.
> - Pre-finalized downgrade path still works with the pinned version.
> - Non-rolling-upgrade coverage for the new layout feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
