Hi, Follow-up to my report below, with new findings that change the question. Cc'ing dm-devel because this now looks like a dm-cache (lvmcache) issue underneath Btrfs. Everything below was read-only; the origin LV and the cache SSD are both blockdev --setro and nothing has been written to either. The NAS vendor's (TerraMaster) support is also involved.
Short version: the detached cache SSD holds the current, checksum-valid copy of every Btrfs metadata block that is stale on the origin, but the cache metadata marks every cached block clean (0 dirty), so a forced uncache dropped them. How the cache was detached -------------------------- LVM metadata history on the origin PV (vg0 on md RAID6), times EDT: - seqno 23, 2026-10-29 21:31:53: "lvconvert --uncache -y --force vg0/lv0" - seqno 24, 2026-10-29 21:31:55: "vgreduce vg0 --removemissing" The cache PV (an md RAID1 on one NVMe) was not assembled after a vendor OS update/restart, and the vendor OS forced the uncache. Cache mode in the LVM metadata: writethrough. Cache metadata (cache_check passes on an image of the cmeta LV) --------------------------------------------------------------- block_size=2048 sectors (1 MiB), nr_cache_blocks=953640, policy=smq, hint_width=4; 953634 mappings, 0 dirty. Damaged Btrfs blocks: origin copy vs cached copy ------------------------------------------------ 64 KiB nodes read from the origin LV and from the cdata area on the SSD (cache block N at cdata + N MiB). crc32c and bytenr checked for every copy. Format gen/owner; both DUP copies agree unless noted. logical wanted origin cached 26935296 917953 910413/3 (copy0) 917953/3 13608045117440 918517 879195/7 918517/8 15797806039040 917945 917909/2 917945/4 (dev tree) 12777307766784 918507 610497/258 918507/2 (extent tree) 12777308094464 918507 608008/258 918507/2 (extent tree) 12777309929472 918507 917901/7 lvl 1 918507/2 lvl 0 13607943929856 918513 604537/260 918513/258 13608042430464 918517 610816/258 918517/260 lvl 2 (subvol root) Every cached copy has a valid checksum and exactly the generation Btrfs expects. The origin copies are valid but much older blocks from other trees. All 20 damaged copies (10 blocks x DUP) are inside cached origin blocks, while overall only 51.4% of metadata blocks (23181/45056) and 5.8% of data blocks were cached. The cache also holds blocks from the final transaction (918517), so it was in use until the end. So for some hot metadata blocks, writes stopped reaching the origin (some since around generation 604537, weeks to months earlier) while reads were served from the cache, and the cache never considered those blocks dirty. Questions --------- 1. For dm-cache: is there a known way a writethrough cache can end up with cached blocks that are newer than the origin but marked clean? (Kernel 6.12.63+ on a TerraMaster F4-424 Pro, TOS 7.0.1341.) 2. Safest recovery: re-create the cache mapping by hand (origin lv0, cmeta, cdata, block size 2048, smq) over dm-snapshot overlays and test a read-only Btrfs mount; or copy the cached metadata blocks onto the origin after testing on an overlay? Anything to watch for, e.g. policy hints, or LVM auto-cleaning the outdated cache PV's metadata if its md/GPT labels are restored (they are currently erased, so LVM cannot see it)? 3. For Btrfs: if the cached versions are put back, is a full btrfs check --readonly plus scrub enough to trust the filesystem, or should I still plan to copy everything off and recreate it? I can provide the cache_dump, the comparison script and output, the LVM metadata text from both PVs and the 97 MiB cmeta/LVM image. Thanks, Aaron On Sat, Oct 31, 2026 at 2:25 AM Aaron Bohenick <[email protected]> wrote: > > Hi, > > I have a single-device Btrfs volume (25.45 TiB, 15.19 TiB used) that no > longer mounts after a series of unclean shutdowns. btrfs restore can read > almost everything, and file data verifies against torrent piece hashes, so > the damage looks confined to metadata. I would like advice on whether the > filesystem can be repaired before anything is written to it. > > Setup > ----- > - TerraMaster F4-424 Pro (TOS 7.0.1341), kernel 6.12.63+ > - 4 x 14 TB HDD, md RAID6 (4/4, resync completed clean, no bitmap) > -> LVM vg0/lv0 -> Btrfs. Profiles: data single, metadata DUP, system DUP > - Until the incident, lv0 had a dm-cache (lvmcache) on an NVMe in > writethrough mode. The NVMe's array was last updated ~7 hours before the > boot failures; after an OS reinstall the cache was no longer attached. > NVMe SMART is clean. > - Features: compat_ro 0x7 (free space tree), incompat 0x361, no > block-group-tree > - btrfs-progs 5.16.2 (vendor) and 7.1 (static release binary) > > Superblock: generation 918517. All four backup roots are gens > 918514-918517; all point to dev root 17014757720064 gen 917953. > > Kernel mount (plain ro, ro,rescue=usebackuproot, ro,rescue=all) > ------------------------------------------------------------------ > parent transid verify failed on logical 26935296 mirror 1 wanted > 917953 found 910413 > parent transid verify failed on logical 13608045117440 mirror 1 wanted > 918517 found 879195 > parent transid verify failed on logical 13608045117440 mirror 2 wanted > 918517 found 879195 > parent transid verify failed on logical 15797806039040 mirror 1 wanted > 917945 found 917909 > parent transid verify failed on logical 15797806039040 mirror 2 wanted > 917945 found 917909 > failed to verify dev extents against chunks: -5 (plain ro) > failed to init dev_stats: -5 (rescue=all) > open_ctree failed: -5 > > 15797806039040 is the DEV_STATS item's block in the dev tree. > > btrfs check --readonly (7.1, also with -b) > ------------------------------------------ > parent transid verify failed on 12777307766784 wanted 918507 found 610497 > parent transid verify failed on 12777308094464 wanted 918507 found 608008 > parent transid verify failed on 12777309929472 wanted 918507 found 917901 > ERROR: child eb corrupted: parent bytenr=13608033124352 item=939 parent > level=1 > child bytenr=12777309929472 child level=1 > ERROR: failed to read block groups: Input/output error > ERROR: cannot open file system > > What works > ---------- > - btrfs restore -i reads 11 of 13 subvolumes (~208,000 files). Two > subvolume roots are bad: 258 (child eb corrupted) and 260 ("root [260 0] > level 0 does not match 2"). A raw scan found intact level-2 roots for > both at gens 918513-918517, but their children are stale too. > - Some fs tree roots are stale in mirror 1 but good in mirror 2, which is > why restore succeeds where dump-tree -t shows the wrong block. > - I restored sample files from 2023 through the day of the incident and > verified 5,687/5,687 torrent pieces, so data extents look intact. > > What I tried (on a dm-snapshot overlay only; origin is blockdev --setro) > ------------------------------------------------------------------------- > - btrfs check --repair --init-extent-tree (7.1): "Creating a new extent > tree", then repeated "unable to find ref byte nr ... parent 0 root 10" > (free space tree blocks) and one for root 1, then "failed to commit > transaction: Read-only file system" (transaction aborted). > - btrfs rescue clear-space-cache v2: fails to open (failed to read block > groups). > (Progs was run in a private mount namespace exposing only the overlay, to > avoid it selecting the origin with the duplicate fsid.) > > Questions > --------- > 1. Is there a sequence that can get past the stale extent tree / free > space tree / dev tree blocks, e.g. dropping the free space tree root and > the DEV_STATS item before --init-extent-tree, or rebuilding the dev tree > from the chunk tree? I can test anything on the overlay first. > 2. Does the pattern (blocks hundreds of thousands of generations stale in > both DUP copies, plus mirror-1-only staleness elsewhere) point to lost > writes below Btrfs, e.g. the dm-cache being detached? > 3. If repair is not realistic, is btrfs restore the right way to copy out > ~15 TiB, or is there a better option? > > I can provide full dump-super -f output, dump-tree of the chunk/root/dev > trees, the restore logs, or try a btrfs-image if that is useful. > > Thanks, > Aaron
