Hi Ming-Hung,
Thanks for the reply! The thin-provisioning-tools are version 0.9.0-2.
Maybe I need a better repository? Here is the sources.list:
deb http://ftp.de.debian.org/debian bookworm main contrib
deb http://ftp.de.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org bookworm-security main contrib
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
The raw metadata is about 17 GB. Would it be better for me to make it
available for download or just let you log into the machine directly?
Thanks,
Ray
------ Original Message ------
From "Ming-Hung Tsai" <[email protected]>
To "Ray Davis" <[email protected]>
Cc [email protected]
Date 10.04.2026 05:43:35
Subject Re: LVM-thin metadata corruption on Proxmox after RAID-5 issues
– thin_repair/thin_dump fail
On Fri, Apr 10, 2026 at 12:08 AM Ray Davis <[email protected]> wrote:
Hello,
I’m looking for advice on an LVM-thin metadata corruption case on a
Proxmox host. I have stopped further repair attempts and preserved the
current state for analysis.
Environment
* Proxmox VE host
* Thin pool: `VMDATA0/VMDATA0`
* Backing storage had RAID-5 issues involving two disks
* After reseating the disks, the RAID came back online, but the thin
pool would no longer activate
Original Proxmox/LVM error
`activating LV 'VMDATA0/VMDATA0' failed: Check of pool VMDATA0/VMDATA0
failed (status:1). Manual repair required!`
What I tried
1. `vgcfgbackup VMDATA0`
2. `lvconvert --repair VMDATA0/VMDATA0`
This failed with:
`value size mismatch: expected 8, but got 24 (block 13182)`
At that point I added temporary VG space on a USB disk so I could try
manual metadata recovery.
Manual recovery steps attempted
* Created temporary metadata LVs
* Used `lvconvert --swapmetadata` to extract the pool metadata
* Activated the extracted metadata LV
* Preserved a raw image of the extracted metadata
* Ran `thin_check`
* Ran `thin_repair`
* Ran `thin_dump —repair`
Current results
`thin_check /dev/VMDATA0/meta_extract` reports:
* `missing devices: [0, -]`
* `bad checksum in btree node (block 79511)`
* `missing all mappings for devices: [0, -]`
* `bad checksum in btree node (block 79506)`
`thin_repair -i /dev/VMDATA0/meta_extract -o /dev/VMDATA0/repair_meta`
fails with:
`value size mismatch: expected 8, but got 24 (block 13182)`
`thin_dump --repair -o /root/VMDATA0_repaired.xml
/dev/VMDATA0/meta_extract` fails with the same:
`value size mismatch: expected 8, but got 24 (block 13182)`
Current LV state at the time of extraction looked like this:
* thin pool `VMDATA0`
* extracted metadata LV `meta_extract`
* fresh target metadata LV `repair_meta`
What I have preserved
* Raw metadata image from the extracted metadata LV:
`VMDATA0_meta_extract.raw`
* A second raw copy from the extracted LV device
* `vgcfgbackup` output and LVM archive files
* Diagnostics bundle with:
* `pvs`, `vgs`, `lvs`
* `thin_check` output
* `dmesg`
* kernel journal
* `lvm version`
* checksums
I can provide access to the full case directory or a tarball over HTTP
if someone is willing to look at it. I would prefer to share the link
privately with anyone interested.
My main questions are:
1. Is there any remaining offline recovery path worth trying with the
standard dm-thin/LVM tools?
2. Does this failure pattern usually indicate that the mapping metadata
is beyond recovery by `thin_repair`/`thin_dump`?
3. Would it be useful to inspect the raw metadata image further, and if
so, what specific tools or commands would you recommend next?
I can provide any command output that would be useful.
Thanks very much for any guidance,
Ray
Hi,
The specific error you mentioned, "value size mismatch: expected 8,
but got 24 (block 13182)", suggests it's an older version of
thin-provisioning-tools. Hopefully, a newer version will help resolve
this.
Would you mind providing the raw metadata image for me to look into? Thank you.
Hank
