Hi, I have configured a single host HCI environment through the GUI wizard in 4.3.9. Initial setup has thai layout of disks, as seen by the operating system: /dev/sda -> for ovirt-node-ng OS /dev/nvme0n1 --> for gluster, engine and data volumes /dev/nvme1n1 --> for gluster, vmstore volume
So far so good and all is ok. I notice that, even with single path internal disks, at the end oVirt configures the gluster disks as multipath devices and LVM2 PV structure on top of the multipath devices. Is this for "code optimization" at low level or what is the rationale for that, as with Gluster normally you do use local disks and so single path? Multipath structure generated: [root@ovirt ~]# multipath -l nvme.8086-50484b53373530353031325233373541474e-494e54454c205353 dm-5 NVME,INTEL SSDPED1K375GA size=349G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active `- 0:0:1:0 nvme0n1 259:0 active undef running eui.01000000010000005cd2e4b5e7db4d51 dm-6 NVME,INTEL SSDPEDKX040T7 size=932G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active `- 2:0:1:0 nvme1n1 259:2 active undef running [root@ovirt ~]# Anyway, on top of the multipah devices On /dev/nvme0n1: gluster_vg_nvme0n1 volume group with gluster_lv_data and gluster_lv_engine On /dev/nvme1n1 gluster_vg_nvme1n1 volume group with gluster_lv_vmstore logical volume The problem arises when I add another nvme disk, that, occupying a PCI slot, it seems has always higher priority of the previous /dev/nvme1n1 disk and so takes its name. After booting the node: old nvme0n1 --> unmodified name old nvme1n1 --> becomes nvme2n1 new disk --> gets name nvme1n1 >From a funcional point of view I have no problems apart LVM warnings I send below and also because the xfs entries in fstab are with UUID: UUID=fa5dd3cb-aeef-470e-b982-432ac896d87a /gluster_bricks/engine xfs inode64,noatime,nodiratime 0 0 UUID=43bed7de-66b1-491d-8055-5b4ef9b0482f /gluster_bricks/data xfs inode64,noatime,nodiratime 0 0 UUID=b81a491c-0a4c-4c11-89d8-9db7fe82888e /gluster_bricks/vmstore xfs inode64,noatime,nodiratime 0 0 lvs commands get: [root@ovirt ~]# lvs WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp. WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl. WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV. WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV. LV VG Attr LSize Pool Origin ... Or, for the old nvme1n1 disk, now nvme2n1 multipath device: [root@ovirt ~]# pvdisplay /dev/mapper/eui.01000000010000005cd2e4e359284f51 WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp. WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl. WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV. WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV. --- Physical volume --- PV Name /dev/mapper/eui.01000000010000005cd2e4e359284f51 VG Name gluster_vg_nvme1n1 PV Size 931.51 GiB / not usable 1.71 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 238467 Free PE 0 Allocated PE 238467 PV UUID O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl [root@ovirt ~]# I'm able to create PV on top of the new multipath device detected by the system (see the nvme1n1 of the underlying disk): eui.01000000010000005cd2e4b5e7db4d51 dm-6 NVME,INTEL SSDPEDKX040T7 size=3.6T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active `- 1:0:1:0 nvme1n1 259:1 active undef running [root@ovirt ~]# pvcreate --dataalignment 256K /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp. WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl. WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV. WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV. Physical volume "/dev/mapper/eui.01000000010000005cd2e4b5e7db4d51" successfully created. [root@ovirt ~]# But then I'm unable to create a VG on top of it: [root@ovirt ~]# vgcreate gluster_vg_4t /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp. WARNING: Not using device /dev/nvme1n1 for PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9. WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl. WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV. WARNING: PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9 prefers device /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 because device is in dm subsystem. WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV. WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp. WARNING: Not using device /dev/nvme1n1 for PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9. WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl. WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because of previous preference. WARNING: PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9 prefers device /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 because of previous preference. WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because of previous preference. Cannot use device /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 with duplicates. [root@ovirt ~]# the same with the "-f" option. I suspect I can solve the problem filtering out /dev/nvme* devices in lvm.conf, but I'm not sure. The OS disk is seen as sda so it should not have problems with this Something like this: filter = [ "r|/dev/nvme|", "a|.*/|" ] And also I am not sure if I have to rebuild the initrd or not in this case and if so what would be the exact sequence of commands to execute. Any suggestions? Thanks in advance, Gianluca
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AGFVU2MCWKXQZNDW277EEUR3YF6DLFJ3/
