For anyone else who is considering using VDO over DRBD. This is what we discovered about getting it to work.
From: Sweet Tea Dorminy <[email protected]> Sent: Tuesday, July 6, 2021 1:50 PM To: vdo-devel <[email protected]> Subject: Getting snapshotting working with VDO: a recent experience. Recently I had the pleasure of helping someone figure out how to use VDO with their snapshotting solution, and figured I'd send out a summary, with their permission, to the list in case it helps anyone else. Setup: Eric was using DRBD for data replication, and VDO on top to provide data reduction, similarly to Linbit's article<https://linbit.com/blog/albireo-virtual-data-optimizer-vdo-on-drbd/>. The stack looked like this: Physical -> LVM -> DRBD -> VDO -> FileSystem Specifically: /dev/sda /dev/vg/lv /dev/drbd0 /dev/vdo0 /my-filesystem (xfs) Problem: Eric wanted to add snapshotting to the mix, taking a snapshot and verifying that it matched the filesystem. Without VDO in the stack, this is a simple matter of taking a snapshot of /dev/vg/lv and mounting the snapshot. With VDO, though, how to get from /dev/vg/lv-snap to a verifiable filesystem? - 'vdo create --device=/dev/vg/lv-snap' doesn't work -- it formats a new VDO, and complains if it finds an already existing vdo. - 'vdo import' doesn't work -- it complains there's already a VDO with the same UUID on the system. - just mounting /dev/vg/lv-snap doesn't work, as it is a vdo, not a filesystem. Solving: In order to start a VDO from storage containing an already formatted VDO, 'vdo import' is necessary, but as said, it reported a UUID collision. After some searching, we realized passing "--uuid=''" (the empty string) to vdo import would import the snapshot VDO and also change the UUID, so it wouldn't collide with the original VDO. For instance: [sweettea@localhost ~]$ sudo vdo create --name vdo0 --device /dev/fedora_localhost-live/vms Creating VDO vdo0 Logical blocks defaulted to 6802574 blocks. The VDO volume can address 26 GB in 13 data slabs, each 2 GB. It can grow to address at most 16 TB of physical storage in 8192 slabs. If a larger maximum size might be needed, use bigger slabs. Starting VDO vdo0 Starting compression on VDO vdo0 VDO instance 0 volume is ready at /dev/mapper/vdo0 [sweettea@localhost ~]$ sudo lvcreate --size 2G --snapshot --name vms_snap /dev/fedora_localhost-live/vms Logical volume "vms_snap" created. [sweettea@localhost ~]$ sudo vdo import --name vdo0-snap --device /dev/fedora_localhost-live/vms_snap Importing VDO vdo0-snap vdo: ERROR - UUID df61bc46-13dc-4091-bdec-4896c744888c already exists in VDO volume(s) stored on /dev/disk/by-id/dm-uuid-LVM-Gexh1cit2vwmcvIf2AAvullg3mWvrnql0SluovtpLXaYGoyPw84zRD4NPmGO3Nu4 [sweettea@localhost ~]$ sudo vdo import --name vdo0-snap --device /dev/fedora_localhost-live/vms_snap --uuid usage: vdo import [-h] -n <volume> --device <devicepath> [--activate {disabled,enabled}] [--blockMapCacheSize <megabytes>] [--blockMapPeriod <period>] [--compression {disabled,enabled}] [--deduplication {disabled,enabled}] [--emulate512 {disabled,enabled}] [--maxDiscardSize <megabytes>] [--uuid <uuid>] [--vdoAckThreads <threadCount>] [--vdoBioRotationInterval <ioCount>] [--vdoBioThreads <threadCount>] [--vdoCpuThreads <threadCount>] [--vdoHashZoneThreads <threadCount>] [--vdoLogicalThreads <threadCount>] [--vdoLogLevel {critical,error,warning,notice,info,debug}] [--vdoPhysicalThreads <threadCount>] [--writePolicy {async,async-unsafe,sync,auto}] [-f <file>] [--logfile <pathname>] [--verbose] vdo import: error: argument --uuid: expected one argument [sweettea@localhost ~]$ sudo vdo import --name vdo0-snap --device /dev/fedora_localhost-live/vms_snap --uuid '' Importing VDO vdo0-snap Starting VDO vdo0-snap Starting compression on VDO vdo0-snap VDO instance 1 volume is ready at /dev/mapper/vdo0-snap (The documentation on --uuid was somewhat confusing, but passing an empty string worked.) However, mounting the resulting snapshot still didn't work. 'journalctl -K' showed these logs from a different attempt: [3714073.665039] kvdo3:dmsetup: underlying device, REQ_FLUSH: supported, REQ_FUA: not supported [3714073.665041] kvdo3:dmsetup: Using write policy async automatically. [3714073.665042] kvdo3:dmsetup: loading device 'snap_vdo0' [3714073.665055] kvdo3:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 [3714073.724597] kvdo3:dmsetup: starting device 'snap_vdo0' [3714073.724607] kvdo3:journalQ: Device was dirty, rebuilding reference counts [3714074.025715] kvdo3:journalQ: Finished reading recovery journal [3714074.032016] kvdo3:journalQ: Highest-numbered recovery journal block has sequence number 4989105, and the highest-numbered usable block is 4989105 [3714074.337654] kvdo3:physQ0: Replaying entries into slab journals for zone 0 [3714074.536446] kvdo3:physQ0: Recreating missing journal entries for zone 0 [3714074.536503] kvdo3:journalQ: Replayed 6365157 journal entries into slab journals [3714074.536504] kvdo3:journalQ: Synthesized 0 missing journal entries [3714074.538423] kvdo3:journalQ: Saving recovery progress [3714074.803255] kvdo3:logQ0: Replaying 2545622 recovery entries into block map [3714077.211236] kvdo3:journalQ: Flushing block map changes [3714077.231360] kvdo3:journalQ: Rebuild complete. [3714077.296258] kvdo3:journalQ: Entering recovery mode [3714077.296460] kvdo3:dmsetup: Setting UDS index target state to online [3714077.296473] kvdo3:dmsetup: device 'snap_vdo0' started [3714077.296474] kvdo3:dmsetup: resuming device 'snap_vdo0' [3714077.296476] kvdo3:dmsetup: device 'snap_vdo0' resumed [3714077.296493] uds: kvdo3:dedupeQ: loading or rebuilding index: dev=/dev/disk/by-id/dm-uuid-LVM-CKGi2ErfgK2rWEYpTJg0OhkXrkf7bRCAIzSDMIJdu5kf0SZFdqgtOySFCarMb1Cl offset=4096 size=2781704192 [3714077.339599] uds: kvdo3:dedupeQ: Using 6 indexing zones for concurrency. [3714077.448849] kvdo3:packerQ: compression is enabled [3714078.142975] uds: kvdo3:dedupeQ: loaded index from chapter 0 through chapter 85 [3714078.227367] uds: kvdo3:dedupeQ: Replaying volume from chapter 3929 through chapter 4952 [3714078.233636] uds: kvdo3:dedupeQ: unexpected index page map update, jumping from 85 to 3929 [3714082.392773] kvdo3:journalQ: Exiting recovery mode [3714134.313962] XFS (dm-8): Filesystem has duplicate UUID be2e762b-6dea-48c4-83ad-08ece1cac43b - can't mount [3714156.566380] uds: kvdo3:dedupeQ: replay changed index page map update from 85 to 4951 [3714320.750730] kvdo3:dmsetup: suspending device 'snap_vdo0' [3714320.751156] uds: dmsetup: beginning save (vcn 4951) [3714321.135314] uds: dmsetup: finished save (vcn 4951) [3714321.135322] kvdo3:dmsetup: device 'snap_vdo0' suspended [3714321.135360] kvdo3:dmsetup: stopping device 'snap_vdo0' [3714321.201264] kvdo3:dmsetup: device 'snap_vdo0' stopped Based on the red line, the remaining problem was a XFS uuid collision, much like the VDO uuid collision on the first attempt to vdo import. Searching revealed this link<https://www.miljan.org/main/2009/11/16/lvm-snapshots-and-xfs/> which suggested either xfs_admin -U generate or mount -o nouuid to change or ignore the uuid of the XFS filesystem. Using these, Eric was able to use the setup fully: taking a snapshot, starting VDO on the snapshot, and mounting the filesystem stored thereon. Hope this helps someone else in the future! Sweet Tea Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
_______________________________________________ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list [email protected] https://lists.linbit.com/mailman/listinfo/drbd-user
