Hi, I experienced a weird Failure. I hope I explain it understandable and not too hard to read
We had issues with our backups, that they can't finalize Here is a Thread someone having the same issue. https://forums.veeam.com/ovirt-kvm-f62/veeam-rhv-12-1-command-removeimageticketvds-failed-t91699.html#p510618 so I made some investigations and recognized, it seems to be an Issue with Snapshot 1. I created a Snapshot https://paste.fo/c0b8e77a3400 what works 2. Then I tried to deleted the snapshot, what didn't work https://paste.fo/013e5632e0d6 So I logged in to a node and checked several things First strange thing was, I can see pvs vgs etc, I am used, that ovirt hides them in the System (on one of those nodes it's still the case) but anyways [root@ovnb05 ~]# pvs WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb. WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to /dev/mapper/360002ac000000000000000880002a7fd). PV VG Fmt Attr PSize PFree /dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-- <12.00t 10.77t /dev/sda3 rl lvm2 a-- 892.16g 0 [unknown] 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-m <12.00t <12.00t So I did [root@ovnb05 ~]# pvdisplay WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb. WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to /dev/mapper/360002ac000000000000000880002a7fd). --- Physical volume --- PV Name /dev/mapper/360002ac000000000000000870002a7fd VG Name 515bebca-972b-42ac-abff-d76af0071613 PV Size 12.00 TiB / not usable 384.00 MiB Allocatable yes PE Size 128.00 MiB Total PE 98301 Free PE 88247 Allocated PE 10054 PV UUID D6Typd-s7lA-PyoI-PvrL-rd9N-mS64-VeV1j3 --- Physical volume --- PV Name [unknown] VG Name 515bebca-972b-42ac-abff-d76af0071613 PV Size 12.00 TiB / not usable 384.00 MiB Allocatable yes PE Size 128.00 MiB Total PE 98301 Free PE 98301 Allocated PE 0 PV UUID uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb --- Physical volume --- PV Name /dev/sda3 VG Name rl PV Size 892.16 GiB / not usable 2.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 228393 Free PE 0 Allocated PE 228393 PV UUID kRe0EF-736L-QwZA-iGtM-67wp-8cr4-DKKcX2 I have never seen an unknown before, so I guess something happened to that LUN But the LUN is available and in /var/log/messages I can't find any issue with the LUN or the path etc. [root@ovnb05 data-center]# blkid |grep uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb /dev/sdi: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdg: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sde: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdn: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdc: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdl: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdr: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdp: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" So I did some nasty stuff, like removing it and trying to add it again, through that process it told me that there is no meta Data on the Device So at the End I am a little bit lost now. Maybe the storage was failing for some reason or maybe veeam messed something up, but what leaves me afraid is, that the Ovirt-Engine didn't show me any Problem with the Cluster or the Storage. I mean it's obvious, that there was/is a huge failure with the storage and Ovirt did not recognize neither did it alarm me. At the moment I detached the LUN from the Storage Domain and created a seperate one (cause I recognized that having several LUN's in a Storage Domain seems to be an bad Idea) and one PVS is now hidden, the other one not [root@ovnb05 data-center]# pvs PV VG Fmt Attr PSize PFree /dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-- <12.00t 11.81t /dev/sda3 rl lvm2 a-- 892.16g 0 [root@ovnb05 data-center]# cat /etc/lvm/devices/system.devices # LVM uses devices listed in this file. # Created by LVM command vgs pid 6123 at Fri Feb 16 18:16:35 2024 VERSION=1.1.7 IDTYPE=mpath_uuid IDNAME=mpath-360002ac000000000000000870002a7fd DEVNAME=/dev/mapper/360002ac000000000000000870002a7fd PVID=D6Typds7lAPyoIPvrLrd9NmS64VeV1j3 IDTYPE=sys_wwid IDNAME=naa.61c721d06b5fb2002c49fbecd02f9d90 DEVNAME=/dev/sda3 PVID=kRe0EF736LQwZAiGtM67wp8cr4DKKcX2 PART=3 [root@ovnb05 data-center]# multipath -ll 360000970000197800382533030303031 dm-5 EMC,SYMMETRIX size=5.6M features='1 queue_if_no_path' hwhandler='0' wp=ro `-+- policy='service-time 0' prio=1 status=active `- 9:0:25:0 sdj 8:144 active ready running 360002ac000000000000000870002a7fd dm-3 3PARdata,VV size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 9:0:8:1 sdh 8:112 active ready running |- 9:0:0:1 sdb 8:16 active ready running |- 9:0:1:1 sdd 8:48 active ready running |- 9:0:2:1 sdf 8:80 active ready running |- 10:0:7:1 sdq 65:0 active ready running |- 10:0:0:1 sdk 8:160 active ready running |- 10:0:1:1 sdm 8:192 active ready running `- 10:0:2:1 sdo 8:224 active ready running 360002ac000000000000000880002a7fd dm-4 3PARdata,VV size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 9:0:8:2 sdi 8:128 active ready running |- 9:0:0:2 sdc 8:32 active ready running |- 9:0:1:2 sde 8:64 active ready running |- 9:0:2:2 sdg 8:96 active ready running |- 10:0:7:2 sdr 65:16 active ready running |- 10:0:0:2 sdl 8:176 active ready running |- 10:0:1:2 sdn 8:208 active ready running `- 10:0:2:2 sdp 8:240 active ready running 360002ac0000000000000008f0002a7fd dm-26 3PARdata,VV size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 10:0:1:3 sdt 65:48 active ready running |- 10:0:0:3 sds 65:32 active ready running |- 10:0:7:3 sdv 65:80 active ready running |- 10:0:2:3 sdu 65:64 active ready running |- 9:0:1:3 sdx 65:112 active ready running |- 9:0:2:3 sdy 65:128 active ready running |- 9:0:0:3 sdw 65:96 active ready running `- 9:0:8:3 sdz 65:144 active ready running If I forgot something or you need more input just let me know, maybe you can light me up what's wrong thx in advance for any help/suggestions _______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/KEWM4EENJ6WVNTZSQLNFA5RVIQTPCMRX/
