[
https://issues.apache.org/jira/browse/HDDS-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-15860:
------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Snapshot defrag fails when directly ingesting a single live RocksDB SST
> -----------------------------------------------------------------------
>
> Key: HDDS-15860
> URL: https://issues.apache.org/jira/browse/HDDS-15860
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> h2. Problem
> Incremental snapshot defragmentation computes RocksDB SST files that differ
> between two snapshot versions and applies the resulting changes to a
> temporary checkpoint database.
> When a table has exactly one candidate SST and the snapshot version is
> greater than zero, {{SnapshotDefragService}} bypasses the normal
> spill-and-rewrite path and passes the live SST directly to
> {{Table.loadFromFile}}.
> The candidate is a live RocksDB SST selected at file granularity, not an
> external SST containing an exact logical bucket delta. The observed behavior
> differs by RocksDB version:
> * With RocksDB 7.7.3 and 10.10.1.1, a DB-generated flush SST is rejected by
> external-file ingestion with {{External file version not found}}.
> DB-generated SSTs lack the {{rocksdb.external_sst_file.version}} and
> {{rocksdb.external_sst_file.global_seqno}} properties, and their internal
> sequence numbers are not valid external-file input.
> * On current master with RocksDB 10.10.1.1, an SST previously ingested into a
> live database can be accepted, but it installs the value stored in that live
> SST instead of the logical value from the current snapshot. With RocksDB
> 7.7.3, reusing such a file can instead fail global-sequence-number validation.
> The direct path also bypasses bucket-prefix filtering,
> current-versus-previous snapshot comparison, and tombstone generation.
> h2. Trigger condition
> * The snapshot version is greater than zero.
> * Exactly one candidate SST is returned for an incremental table.
> Once triggered, snapshot defragmentation cannot correctly advance the
> affected snapshot to its next defragmented version.
> h2. Proposed fix
> Remove the single-file direct-ingestion shortcut. All candidate SST sets,
> including singleton sets, should pass through {{spillTableDiffIntoSstFile}}.
> This existing path:
> * Filters keys by the target bucket prefix.
> * Compares values in the current and previous snapshots.
> * Emits only actual updates and insertions.
> * Emits tombstones for deletions.
> * Creates a fresh external SST suitable for RocksDB ingestion.
> h2. Why existing tests did not catch this
> The incremental-defragmentation control flow was covered, but the RocksDB
> ingestion boundary was mocked:
> * The original background-defrag change did not exercise real incremental SST
> ingestion.
> * {{testPerformIncrementalDefragmentation}} creates empty placeholder
> {{.sst}} files and mocks {{SstFileSetReader}}, {{RDBSstFileWriter}}, and
> {{Table.loadFromFile}}. For snapshot versions greater than zero, it
> explicitly expected the single-file table to be passed directly to the mocked
> loader, so the test codified the unsafe shortcut without invoking RocksDB JNI.
> * Lower-level {{Table.loadFromFile}} tests first create a valid external SST
> with {{dumpToFileWithPrefix}}. They verify external-SST ingestion, not
> ingestion of a live DB-generated SST.
> * The snapshot-defrag admin integration test verifies that the CLI can
> trigger the service on OM nodes. It does not create snapshot data or verify
> incremental SST ingestion and the resulting checkpoint contents.
> As a result, no existing test combined a nonzero snapshot version, exactly
> one live candidate SST, and a real {{Table.loadFromFile}} call.
> h2. Test coverage
> * Update the existing incremental-defragmentation unit test to expect
> rewriting for every snapshot version and candidate-file count.
> * Add a real RocksDB regression test covering both DB-generated live SSTs and
> SSTs previously ingested into a live database.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
