[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 04, 2022 at 04:02:56PM +0200, Nir Soffer wrote: > On Fri, Feb 4, 2022 at 3:49 PM Richard W.M. Jones wrote: > > > > On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote: > > > Can be fixed with: > > > > > > $ sudo chcon -R -t nfs_t mnt > > > > Yes that did work, thanks. > > Warning: that this configuration is a trap - if you add another host > to this system, > the system will try to mount the same device (/dev/sdb1) on the new > host. Since the > other host does not have the same disk, the mount will fail, and the > other host will > be deactivated, since it cannot access all storage. Point taken! This is only a test however. Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=2050721 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5G7FB3DDW77WKVPLS2LL325RQHORG6WY/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 4, 2022 at 3:18 PM Richard W.M. Jones wrote:
>
> On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
> > On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones
> > wrote:
> > >
> > > On Fri, Feb 04, 2022 at 08:42:08AM +, Richard W.M. Jones wrote:
> > > > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> > > > > This is expected on oVirt, our multipath configuration is
> > > > > intentionally grabbing
> > > > > any device that multipath can work with, even if the device only has
> > > > > one path.
> > > > > The motivation is to be able to configure a system when only one path
> > > > > is
> > > > > available (maybe you have an hba/network/server issue), and once the
> > > > > other
> > > > > paths are available the system will use them transparently.
> > > > >
> > > > > To avoid this issue with local devices, you need to blacklist the
> > > > > device.
> > > > >
> > > > > Add this file:
> > > > >
> > > > > $ cat /etc/multipath/conf.d/local.conf
> > > > > blacklist {
> > > > > wwid "QEMU HARDDISK"
> > > > > }
> > > >
> > > > Thanks - for the mailing list record the syntax that worked for me is:
> > > >
> > > > # cat /etc/multipath/conf.d/local.conf
> > > > blacklist {
> > > > wwid ".*QEMU_HARDDISK.*"
> > > > }
> > > >
> > > > > Configuring NFS on some other machine is easy.
> > > > >
> > > > > I'm using another VM for this, so I can easily test negative flows
> > > > > like stopping
> > > > > or restarting the NFS server while it is being used by vms or storage
> > > > > operations.
> > > > > I'm using 2G alpine vm for this, it works fine even with 1G memory.
> > > >
> > > > I think I can get local storage working now (I had it working before).
> > >
> > > Well finally it fails with:
> > >
> > > 2022-02-04 09:14:55,779Z ERROR
> > > [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand]
> > > (default task-2) [25a32edf] Command
> > > 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand'
> > > failed: EngineException:
> > > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> > > VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS,
> > > error = Could not initialize cluster lock: (), code = 701 (Failed with
> > > error unexpected and code 16)
> >
> > The error "Could not initialize cluster lock" comes from vdsm. Usually
> > engine log is
> > not the best way to debug such failures. This is only the starting
> > point and you need to
> > go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
>
> I can't really see anything relevant in supervdsm.log, it's all fairly
> neutral debug messages.
>
> > Since this error
> > comes from sanlock, we also may have useful info in /var/log/sanlock.log.
>
> Interesting:
>
> 2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open
> /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids
> 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group
> sanlock 179 has access to disk or file.
>
> I think it's quite likely that the sanlock daemon does not have access
> here, since (see below) I choown'd the root of the xfs filesystem to
> 36:36 (otherwise vdsm complains).
>
> > Can you share instructions on how to reproduce this issue?
>
> I have one engine and one node (both happen to be VMs, but I don't
> believe that is relevant here). It's running Version 4.4.10.6-1.el8.
>
> I added a second disk to the node, and disabled multipath as
> previously discussed. The second disk is /dev/sdb1. I formatted it
> as xfs and chowned the root of the filesystem to 36:36.
>
> In the admin portal, Storage -> Domains -> New domain
>
> Storage type: Posix compliant fs
>
> Name: ovirt-data
>
> Path: /dev/sdb1
>
> VFS type: xfs
>
> Hit OK ->
> Error while executing action AddPosixFsStorageDomain: Unexpected exception
Avihai, are we testing this configuration?
I'm not sure this is useful for real users - you need to have the same
device available
on multipath hosts, backed up by a shared file system, like GFS2.
[1] https://en.wikipedia.org/wiki/GFS2
Nir
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WYY3GGAQ2JHG5Z5EEURQ5AZ5JP3AE32R/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 4, 2022 at 3:49 PM Richard W.M. Jones wrote: > > On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote: > > Can be fixed with: > > > > $ sudo chcon -R -t nfs_t mnt > > Yes that did work, thanks. Warning: that this configuration is a trap - if you add another host to this system, the system will try to mount the same device (/dev/sdb1) on the new host. Since the other host does not have the same disk, the mount will fail, and the other host will be deactivated, since it cannot access all storage. Nir ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7BZBMO5TXNKLBXVGKAUHBGRJAS7VWUHS/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 4, 2022 at 3:49 PM Richard W.M. Jones wrote: > > On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote: > > Can be fixed with: > > > > $ sudo chcon -R -t nfs_t mnt > > Yes that did work, thanks. > > Is this still a bug? For NFS this works out of the box - I don't think you need to relebal anything manually. As a user I would expect that mounting posix file system will also work out of the box. For example vdsm ca relabel the mount point when creating a storage domain. Also I'm not sure using nfs_t is the right label to use, we better discuss this with selinux folks. So I think this issue is worth a bug, in the worst case we will close it, and the bug will also document the workaround. Nir ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WUXD2DQ6DDCVOK2HNMEOHLJNRWXKWOE2/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote: > Can be fixed with: > > $ sudo chcon -R -t nfs_t mnt Yes that did work, thanks. Is this still a bug? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q2E35J2MSA36DP3A5TJIKIMWJP4PM52D/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 4, 2022 at 3:18 PM Richard W.M. Jones wrote:
>
> On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
> > On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones
> > wrote:
> > >
> > > On Fri, Feb 04, 2022 at 08:42:08AM +, Richard W.M. Jones wrote:
> > > > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> > > > > This is expected on oVirt, our multipath configuration is
> > > > > intentionally grabbing
> > > > > any device that multipath can work with, even if the device only has
> > > > > one path.
> > > > > The motivation is to be able to configure a system when only one path
> > > > > is
> > > > > available (maybe you have an hba/network/server issue), and once the
> > > > > other
> > > > > paths are available the system will use them transparently.
> > > > >
> > > > > To avoid this issue with local devices, you need to blacklist the
> > > > > device.
> > > > >
> > > > > Add this file:
> > > > >
> > > > > $ cat /etc/multipath/conf.d/local.conf
> > > > > blacklist {
> > > > > wwid "QEMU HARDDISK"
> > > > > }
> > > >
> > > > Thanks - for the mailing list record the syntax that worked for me is:
> > > >
> > > > # cat /etc/multipath/conf.d/local.conf
> > > > blacklist {
> > > > wwid ".*QEMU_HARDDISK.*"
> > > > }
> > > >
> > > > > Configuring NFS on some other machine is easy.
> > > > >
> > > > > I'm using another VM for this, so I can easily test negative flows
> > > > > like stopping
> > > > > or restarting the NFS server while it is being used by vms or storage
> > > > > operations.
> > > > > I'm using 2G alpine vm for this, it works fine even with 1G memory.
> > > >
> > > > I think I can get local storage working now (I had it working before).
> > >
> > > Well finally it fails with:
> > >
> > > 2022-02-04 09:14:55,779Z ERROR
> > > [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand]
> > > (default task-2) [25a32edf] Command
> > > 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand'
> > > failed: EngineException:
> > > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> > > VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS,
> > > error = Could not initialize cluster lock: (), code = 701 (Failed with
> > > error unexpected and code 16)
> >
> > The error "Could not initialize cluster lock" comes from vdsm. Usually
> > engine log is
> > not the best way to debug such failures. This is only the starting
> > point and you need to
> > go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
>
> I can't really see anything relevant in supervdsm.log, it's all fairly
> neutral debug messages.
>
> > Since this error
> > comes from sanlock, we also may have useful info in /var/log/sanlock.log.
>
> Interesting:
>
> 2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open
> /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids
> 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group
> sanlock 179 has access to disk or file.
The issue is selinux:
NFS domain:
$ ls -lhZ
/rhev/data-center/mnt/alpine\:_01/e9467633-ee31-4e15-b3f8-3812b374c764/dom_md/
total 2.3M
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 1.0M Feb 4 15:32 ids
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Jan 20 23:53 inbox
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 2.0M Jan 20 23:54 leases
-rw-r--r--. 1 vdsm kvm system_u:object_r:nfs_t:s0 354 Jan 20 23:54 metadata
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Jan 20 23:53 outbox
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 1.3M Jan 20 23:53 xleases
The posix domain mount (mounted manually):
$ ls -lhZ mnt/689c22c4-e264-4873-aa75-1aa4970d4366/dom_md/
total 252K
-rw-rw. 1 vdsm kvm system_u:object_r:unlabeled_t:s00 Feb 4 15:23 ids
-rw-rw. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 16M Feb 4 15:23 inbox
-rw-rw. 1 vdsm kvm system_u:object_r:unlabeled_t:s00 Feb 4 15:23 leases
-rw-r--r--. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 316 Feb 4
15:23 metadata
-rw-rw. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 16M Feb 4 15:23 outbox
-rw-rw. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 1.3M Feb 4
15:23 xleases
Can be fixed with:
$ sudo chcon -R -t nfs_t mnt
$ ls -lhZ mnt/689c22c4-e264-4873-aa75-1aa4970d4366/dom_md/
total 252K
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s00 Feb 4 15:23 ids
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Feb 4 15:23 inbox
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s00 Feb 4 15:23 leases
-rw-r--r--. 1 vdsm kvm system_u:object_r:nfs_t:s0 316 Feb 4 15:23 metadata
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Feb 4 15:23 outbox
-rw-rw. 1 vdsm kvm system_u:object_r:nfs_t:s0 1.3M Feb 4 15:23 xleases
After this change delete the storage domain directory:
$ rm -rf mnt/689c22c4-e264-4873-aa75-1aa4970d4366
Since vdsm will refuse to create a storage domain in a
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 4, 2022 at 3:18 PM Richard W.M. Jones wrote:
>
> On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
> > On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones
> > wrote:
> > >
> > > On Fri, Feb 04, 2022 at 08:42:08AM +, Richard W.M. Jones wrote:
> > > > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> > > > > This is expected on oVirt, our multipath configuration is
> > > > > intentionally grabbing
> > > > > any device that multipath can work with, even if the device only has
> > > > > one path.
> > > > > The motivation is to be able to configure a system when only one path
> > > > > is
> > > > > available (maybe you have an hba/network/server issue), and once the
> > > > > other
> > > > > paths are available the system will use them transparently.
> > > > >
> > > > > To avoid this issue with local devices, you need to blacklist the
> > > > > device.
> > > > >
> > > > > Add this file:
> > > > >
> > > > > $ cat /etc/multipath/conf.d/local.conf
> > > > > blacklist {
> > > > > wwid "QEMU HARDDISK"
> > > > > }
> > > >
> > > > Thanks - for the mailing list record the syntax that worked for me is:
> > > >
> > > > # cat /etc/multipath/conf.d/local.conf
> > > > blacklist {
> > > > wwid ".*QEMU_HARDDISK.*"
> > > > }
> > > >
> > > > > Configuring NFS on some other machine is easy.
> > > > >
> > > > > I'm using another VM for this, so I can easily test negative flows
> > > > > like stopping
> > > > > or restarting the NFS server while it is being used by vms or storage
> > > > > operations.
> > > > > I'm using 2G alpine vm for this, it works fine even with 1G memory.
> > > >
> > > > I think I can get local storage working now (I had it working before).
> > >
> > > Well finally it fails with:
> > >
> > > 2022-02-04 09:14:55,779Z ERROR
> > > [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand]
> > > (default task-2) [25a32edf] Command
> > > 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand'
> > > failed: EngineException:
> > > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> > > VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS,
> > > error = Could not initialize cluster lock: (), code = 701 (Failed with
> > > error unexpected and code 16)
> >
> > The error "Could not initialize cluster lock" comes from vdsm. Usually
> > engine log is
> > not the best way to debug such failures. This is only the starting
> > point and you need to
> > go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
>
> I can't really see anything relevant in supervdsm.log, it's all fairly
> neutral debug messages.
>
> > Since this error
> > comes from sanlock, we also may have useful info in /var/log/sanlock.log.
>
> Interesting:
>
> 2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open
> /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids
> 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group
> sanlock 179 has access to disk or file.
>
> I think it's quite likely that the sanlock daemon does not have access
> here, since (see below) I choown'd the root of the xfs filesystem to
> 36:36 (otherwise vdsm complains).
>
> > Can you share instructions on how to reproduce this issue?
>
> I have one engine and one node (both happen to be VMs, but I don't
> believe that is relevant here). It's running Version 4.4.10.6-1.el8.
>
> I added a second disk to the node, and disabled multipath as
> previously discussed. The second disk is /dev/sdb1. I formatted it
> as xfs and chowned the root of the filesystem to 36:36.
Looks right, forgetting to change ownership is a common mistake.
> In the admin portal, Storage -> Domains -> New domain
>
> Storage type: Posix compliant fs
>
> Name: ovirt-data
>
> Path: /dev/sdb1
>
> VFS type: xfs
>
> Hit OK ->
> Error while executing action AddPosixFsStorageDomain: Unexpected exception
I reproduce with vdsm-4.50.0.5-1.el8.x86_64. on RHEL 8.6 nightly.
Can you file a oVirt/vdsm bug for this?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TXPJHOTW4FN7FNLC3WHZP3H5DRKPGOWD/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
> On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones wrote:
> >
> > On Fri, Feb 04, 2022 at 08:42:08AM +, Richard W.M. Jones wrote:
> > > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> > > > This is expected on oVirt, our multipath configuration is intentionally
> > > > grabbing
> > > > any device that multipath can work with, even if the device only has
> > > > one path.
> > > > The motivation is to be able to configure a system when only one path is
> > > > available (maybe you have an hba/network/server issue), and once the
> > > > other
> > > > paths are available the system will use them transparently.
> > > >
> > > > To avoid this issue with local devices, you need to blacklist the
> > > > device.
> > > >
> > > > Add this file:
> > > >
> > > > $ cat /etc/multipath/conf.d/local.conf
> > > > blacklist {
> > > > wwid "QEMU HARDDISK"
> > > > }
> > >
> > > Thanks - for the mailing list record the syntax that worked for me is:
> > >
> > > # cat /etc/multipath/conf.d/local.conf
> > > blacklist {
> > > wwid ".*QEMU_HARDDISK.*"
> > > }
> > >
> > > > Configuring NFS on some other machine is easy.
> > > >
> > > > I'm using another VM for this, so I can easily test negative flows like
> > > > stopping
> > > > or restarting the NFS server while it is being used by vms or storage
> > > > operations.
> > > > I'm using 2G alpine vm for this, it works fine even with 1G memory.
> > >
> > > I think I can get local storage working now (I had it working before).
> >
> > Well finally it fails with:
> >
> > 2022-02-04 09:14:55,779Z ERROR
> > [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand]
> > (default task-2) [25a32edf] Command
> > 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand'
> > failed: EngineException:
> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> > VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS,
> > error = Could not initialize cluster lock: (), code = 701 (Failed with
> > error unexpected and code 16)
>
> The error "Could not initialize cluster lock" comes from vdsm. Usually
> engine log is
> not the best way to debug such failures. This is only the starting
> point and you need to
> go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
I can't really see anything relevant in supervdsm.log, it's all fairly
neutral debug messages.
> Since this error
> comes from sanlock, we also may have useful info in /var/log/sanlock.log.
Interesting:
2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open
/rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids
2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group
sanlock 179 has access to disk or file.
I think it's quite likely that the sanlock daemon does not have access
here, since (see below) I choown'd the root of the xfs filesystem to
36:36 (otherwise vdsm complains).
> Can you share instructions on how to reproduce this issue?
I have one engine and one node (both happen to be VMs, but I don't
believe that is relevant here). It's running Version 4.4.10.6-1.el8.
I added a second disk to the node, and disabled multipath as
previously discussed. The second disk is /dev/sdb1. I formatted it
as xfs and chowned the root of the filesystem to 36:36.
In the admin portal, Storage -> Domains -> New domain
Storage type: Posix compliant fs
Name: ovirt-data
Path: /dev/sdb1
VFS type: xfs
Hit OK ->
Error while executing action AddPosixFsStorageDomain: Unexpected exception
> > I think this feature (local storage) no longer works.
>
> This is not local storage, local storage is a storage domain using a
> local directory
> on a host. This works only when creating a local data center,
> basically each host
> has its own data center.
>
> Nir
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CSKCHQVWWD7EW3A4HPA37RJTKACFATO6/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones wrote:
>
> On Fri, Feb 04, 2022 at 08:42:08AM +, Richard W.M. Jones wrote:
> > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> > > This is expected on oVirt, our multipath configuration is intentionally
> > > grabbing
> > > any device that multipath can work with, even if the device only has one
> > > path.
> > > The motivation is to be able to configure a system when only one path is
> > > available (maybe you have an hba/network/server issue), and once the other
> > > paths are available the system will use them transparently.
> > >
> > > To avoid this issue with local devices, you need to blacklist the device.
> > >
> > > Add this file:
> > >
> > > $ cat /etc/multipath/conf.d/local.conf
> > > blacklist {
> > > wwid "QEMU HARDDISK"
> > > }
> >
> > Thanks - for the mailing list record the syntax that worked for me is:
> >
> > # cat /etc/multipath/conf.d/local.conf
> > blacklist {
> > wwid ".*QEMU_HARDDISK.*"
> > }
> >
> > > Configuring NFS on some other machine is easy.
> > >
> > > I'm using another VM for this, so I can easily test negative flows like
> > > stopping
> > > or restarting the NFS server while it is being used by vms or storage
> > > operations.
> > > I'm using 2G alpine vm for this, it works fine even with 1G memory.
> >
> > I think I can get local storage working now (I had it working before).
>
> Well finally it fails with:
>
> 2022-02-04 09:14:55,779Z ERROR
> [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand]
> (default task-2) [25a32edf] Command
> 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand'
> failed: EngineException:
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS,
> error = Could not initialize cluster lock: (), code = 701 (Failed with error
> unexpected and code 16)
The error "Could not initialize cluster lock" comes from vdsm. Usually
engine log is
not the best way to debug such failures. This is only the starting
point and you need to
go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
Since this error
comes from sanlock, we also may have useful info in /var/log/sanlock.log.
Can you share instructions on how to reproduce this issue?
> I think this feature (local storage) no longer works.
This is not local storage, local storage is a storage domain using a
local directory
on a host. This works only when creating a local data center,
basically each host
has its own data center.
Nir
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SO6SUNKEIGGTJSOD4QQ7DSJ5667C45TZ/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Fri, Feb 04, 2022 at 08:42:08AM +, Richard W.M. Jones wrote:
> On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> > This is expected on oVirt, our multipath configuration is intentionally
> > grabbing
> > any device that multipath can work with, even if the device only has one
> > path.
> > The motivation is to be able to configure a system when only one path is
> > available (maybe you have an hba/network/server issue), and once the other
> > paths are available the system will use them transparently.
> >
> > To avoid this issue with local devices, you need to blacklist the device.
> >
> > Add this file:
> >
> > $ cat /etc/multipath/conf.d/local.conf
> > blacklist {
> > wwid "QEMU HARDDISK"
> > }
>
> Thanks - for the mailing list record the syntax that worked for me is:
>
> # cat /etc/multipath/conf.d/local.conf
> blacklist {
> wwid ".*QEMU_HARDDISK.*"
> }
>
> > Configuring NFS on some other machine is easy.
> >
> > I'm using another VM for this, so I can easily test negative flows like
> > stopping
> > or restarting the NFS server while it is being used by vms or storage
> > operations.
> > I'm using 2G alpine vm for this, it works fine even with 1G memory.
>
> I think I can get local storage working now (I had it working before).
Well finally it fails with:
2022-02-04 09:14:55,779Z ERROR
[org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand]
(default task-2) [25a32edf] Command
'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand'
failed: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error
= Could not initialize cluster lock: (), code = 701 (Failed with error
unexpected and code 16)
I think this feature (local storage) no longer works.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OD43QV4C5HCQGDQVTA72QQBTTU7T6N45/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
> This is expected on oVirt, our multipath configuration is intentionally
> grabbing
> any device that multipath can work with, even if the device only has one path.
> The motivation is to be able to configure a system when only one path is
> available (maybe you have an hba/network/server issue), and once the other
> paths are available the system will use them transparently.
>
> To avoid this issue with local devices, you need to blacklist the device.
>
> Add this file:
>
> $ cat /etc/multipath/conf.d/local.conf
> blacklist {
> wwid "QEMU HARDDISK"
> }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf
blacklist {
wwid ".*QEMU_HARDDISK.*"
}
> Configuring NFS on some other machine is easy.
>
> I'm using another VM for this, so I can easily test negative flows like
> stopping
> or restarting the NFS server while it is being used by vms or storage
> operations.
> I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AU5UUXFNGPMBGETPZI3R445FFWESUGXK/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Thu, Feb 3, 2022 at 3:42 PM Richard W.M. Jones wrote:
>
> On Thu, Feb 03, 2022 at 03:07:20PM +0200, Nir Soffer wrote:
> > On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones wrote:
> > >
> > >
> > > I'm following the instructions here:
> > > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-preparing_and_adding_posix_compliant_file_system_storage
> > >
> > > I've also added an extra virtual disk to my host node which appears as
> > > /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created.
> > > Is udev broken in oVirt node?
> > >
> > > I cannot see anywhere in the dialog where you specify the name of the
> > > device (eg. "/dev/sdb1"). So how's it supposed to work?
> > >
> > > It doesn't work, giving an information-free error message:
> > >
> > > Error while executing action Add Storage Connection: Problem while
> > > trying to mount target
> >
> > You can find more info on the failure in:
> > /var/log/vdsm/supervdsmd.log
>
> vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'xfs',
> '/srv', '/rhev/data-center/mnt/_srv'] failed with rc=32 out=b'' err=b'mount:
> /rhev/data-center/mnt/_srv: /srv is not a block device.\n'
>
> I suppose it expects the name of the block device (ie. /dev/sdb)
> rather than the mount point there.
>
> It also turns out the new device has been "captured" by multipathd:
>
> # multipath -ll
> 0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1 dm-0 QEMU,QEMU HARDDISK
> size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
> `-+- policy='service-time 0' prio=1 status=active
> `- 0:0:0:1 sdb 8:16 active ready running
>
> I've so far not found a way to disable multipathd effectively. Even
> stopping and disabling the service and rebooting doesn't help so I
> guess something starts it up.
This is expected on oVirt, our multipath configuration is intentionally grabbing
any device that multipath can work with, even if the device only has one path.
The motivation is to be able to configure a system when only one path is
available (maybe you have an hba/network/server issue), and once the other
paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf
blacklist {
wwid "QEMU HARDDISK"
}
And run (as root):
multipathd reconfigure
At this point lsblk will show the expected /dev/sdb1 and multipath
will never use this device again.
Adding a serial to the device in libvirt xml will make it easier to blacklist.
>
> > Posix compliant is basically NFS without some mount options:
> > https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca97249247/lib/vdsm/storage/storageServer.py#L174
> >
> > Using a local device on a host is less tested path, I'm not QE is testing
> > this (Avihai, please correct me if you do).
> >
> > If you have multiple hosts, this will break if the local device does not
> > have
> > the same name on all hosts (so using /dev/sdb1 is very fragile). If you have
> > one host it can be fine.
> >
> > Any reason to add a device to the vm, instead of using an NFS server?
> >
> > I guess that your purpose is testing virt-v2v with oVirt, so you want to
> > test
> > a common configuration; NFS is very common for oVirt users.
>
> I don't have an NFS server to use for this.
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping
or restarting the NFS server while it is being used by vms or storage
operations.
I'm using 2G alpine vm for this, it works fine even with 1G memory.
Nir
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EHXYNOUEI5ETO57ZNLKBJY76S5IVYFO2/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Thu, Feb 3, 2022 at 1:30 PM Richard W.M. Jones wrote: > > I'm following the instructions here: > > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-preparing_and_adding_posix_compliant_file_system_storage > > I've also added an extra virtual disk to my host node which appears as > /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. > Is udev broken in oVirt node? > > As you write the term "local", if you plan to have only one host and not add any other in the future for this environment, you had better creating a DC/Cluster of type "Local" (versus "Shared") and then a storage domain of type "Local on Host", where you give it the path where you mounted your additional disk (with correct permissions). I've been using it for many months on one environment with only one very big server (1,5Tb of ram and 2 NVME disks, each 2Tb in size), currently with 36 VMs running without problem. Now at 4.4.10 async 2. The host is now based on Rocky Linux 8.5 (previously CentOS 8.4) with a standalone engine, also that one now based on Rocky Linux 8.5) that is a VM on an external vSphere environment. This same environment previously was based on a single host gluster configuration and I converted it to this configuration without loss of VMs/data (I detached and then imported the previously gluster based fs). HIH, Gianluca ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y62VI36WEFNJR57JAZPCMD66PI6VWDVD/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Thu, Feb 03, 2022 at 03:07:20PM +0200, Nir Soffer wrote: > On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones wrote: > > > > > > I'm following the instructions here: > > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-preparing_and_adding_posix_compliant_file_system_storage > > > > I've also added an extra virtual disk to my host node which appears as > > /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. > > Is udev broken in oVirt node? > > > > I cannot see anywhere in the dialog where you specify the name of the > > device (eg. "/dev/sdb1"). So how's it supposed to work? > > > > It doesn't work, giving an information-free error message: > > > > Error while executing action Add Storage Connection: Problem while trying > > to mount target > > You can find more info on the failure in: > /var/log/vdsm/supervdsmd.log vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'xfs', '/srv', '/rhev/data-center/mnt/_srv'] failed with rc=32 out=b'' err=b'mount: /rhev/data-center/mnt/_srv: /srv is not a block device.\n' I suppose it expects the name of the block device (ie. /dev/sdb) rather than the mount point there. It also turns out the new device has been "captured" by multipathd: # multipath -ll 0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1 dm-0 QEMU,QEMU HARDDISK size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 0:0:0:1 sdb 8:16 active ready running I've so far not found a way to disable multipathd effectively. Even stopping and disabling the service and rebooting doesn't help so I guess something starts it up. > Posix compliant is basically NFS without some mount options: > https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca97249247/lib/vdsm/storage/storageServer.py#L174 > > Using a local device on a host is less tested path, I'm not QE is testing > this (Avihai, please correct me if you do). > > If you have multiple hosts, this will break if the local device does not have > the same name on all hosts (so using /dev/sdb1 is very fragile). If you have > one host it can be fine. > > Any reason to add a device to the vm, instead of using an NFS server? > > I guess that your purpose is testing virt-v2v with oVirt, so you want to test > a common configuration; NFS is very common for oVirt users. I don't have an NFS server to use for this. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TXOKEW3KY3IEB32SMK2OFRORN6WRSYY3/
[ovirt-users] Re: Unclear how to add local (POSIX) storage
On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones wrote: > > > I'm following the instructions here: > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-preparing_and_adding_posix_compliant_file_system_storage > > I've also added an extra virtual disk to my host node which appears as > /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. > Is udev broken in oVirt node? > > I cannot see anywhere in the dialog where you specify the name of the > device (eg. "/dev/sdb1"). So how's it supposed to work? > > It doesn't work, giving an information-free error message: > > Error while executing action Add Storage Connection: Problem while trying > to mount target You can find more info on the failure in: /var/log/vdsm/supervdsmd.log Posix compliant is basically NFS without some mount options: https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca97249247/lib/vdsm/storage/storageServer.py#L174 Using a local device on a host is less tested path, I'm not QE is testing this (Avihai, please correct me if you do). If you have multiple hosts, this will break if the local device does not have the same name on all hosts (so using /dev/sdb1 is very fragile). If you have one host it can be fine. Any reason to add a device to the vm, instead of using an NFS server? I guess that your purpose is testing virt-v2v with oVirt, so you want to test a common configuration; NFS is very common for oVirt users. Nir ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NPJMTKRFVAXPDSJCAFJSYWUE4NS4AS6S/
