[ovirt-users] Manual transfer of VMs from DC to DC
Hi all, Is there any way of transferring VMs manually from DC to DC, without the DCs having connectivity with each other? I was thinking to backup all the export domain directory, and then later rsync this directory VMs to a new NFS share, then import this NFS share as an export domain on the other DC. What do you think? Thanx, Alex ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Recommended gluster setup
Hi all, I see in the ovirt guides that a gluster volume replica 3 with 1 arbiter is recommended. Why not simple replica 3? Is it due to the higher replication data that would cause performance issues? What I am observing is that a VM running on the server which has the arbiter brick has slower read performance then when the same VM runs on another server with a normal brick. Has anyone observed this? Is it because the arbiter does not have the real data on it? Thanx, Alex ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster
Hi all, Seems that I am seeing light in he tunnel! I have done the below: Added the option /etc/glusterfs/glusterd.vol : option rpc-auth-allow-insecure on restarted glusterd Then set the volume option: gluster volume set vms server.allow-insecure on Then disabled the option: gluster volume set vms performance.readdir-ahead off which seems to have been enabled when desperately testing gluster options. then I added again the storage domain with glusterfs (not NFS). This lead me to have really high performance boost on writes of VMs. The dd went from 10MB/s to 60MB/s! And the VM disk benchmarks from few MB to 80 - 100MB/s! Seems that the above two options which were missing did the trick for the gluster. When checking the VM XML I still see the disk defines as file and not network. I am not sure that ligfapi is used from ovirt. I will set also a new VM just in case to check the XML again. Is there any way to confirm use of ligfapi? On Fri, Sep 8, 2017 at 10:08 AM, Abi Askushi wrote: > I don't see any other bottleneck. CPUs are quite idle. Seems that the load > is mostly due to high latency on IO. > > Reading further the gluster docs: > > https://github.com/gluster/glusterfs-specs/blob/master/ > done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md > > I see that I am missing the following options: > > /etc/glusterfs/glusterd.vol : > option rpc-auth-allow-insecure on > > gluster volume set vms gluster server.allow-insecure on > > It says that the above allow qemu to use libgfapi. > When checking the VM XML, I don't see any gluster protocol at the disk > drive: > > >io='threads'/> > > > 222a1312-5efa-4731-8914-9a9d24dccba5 > > > > > > > While at gluster docs it mentions the below type of disk: > > > > > > > > function='0x0'/> > > > > Does the above indicate that ovirt/qemu is not using libgfapi but FUSE only? > This could be the reason of such slow perf. > > > On Thu, Sep 7, 2017 at 1:47 PM, Yaniv Kaul wrote: > >> >> >> On Thu, Sep 7, 2017 at 12:52 PM, Abi Askushi >> wrote: >> >>> >>> >>> On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul wrote: >>> >>>> >>>> >>>> On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul wrote: >>>> >>>>> >>>>> >>>>> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi >>>>> wrote: >>>>> >>>>>> For a first idea I use: >>>>>> >>>>>> dd if=/dev/zero of=testfile bs=1GB count=1 >>>>>> >>>>> >>>>> This is an incorrect way to test performance, for various reasons: >>>>> 1. You are not using oflag=direct , thus not using DirectIO, but using >>>>> cache. >>>>> 2. It's unrealistic - it is very uncommon to write large blocks of >>>>> zeros (sometimes during FS creation or wiping). Certainly not 1GB >>>>> 3. It is a single thread of IO - again, unrealistic for VM's IO. >>>>> >>>>> I forgot to mention that I include oflag=direct in my tests. I agree >>> though that dd is not the correct way to test, hence I mentioned I just use >>> it to get a first feel. More tests are done within the VM benchmarking its >>> disk IO (with tools like IOmeter). >>> >>> I suggest using fio and such. See https://github.com/pcuzner/fio-tools >>>>> for example. >>>>> >>>> Do you have any recommended config file to use for VM workload? >>> >> >> Desktops and Servers VMs behave quite differently, so not really. But the >> 70/30 job is typically a good baseline. >> >> >>> >>> >>>>> >>>>>> >>>>>> When testing on the gluster mount point using above command I hardly >>>>>> get 10MB/s. (On the same time the network traffic hardly reaches >>>>>> 100Mbit). >>>>>> >>>>>> When testing our of the gluster (for example at /root) I get 600 - >>>>>> 700MB/s. >>>>>> >>>>> >>>>> That's very fast - from 4 disks doing RAID5? Impressive (unless you >>>>> use caching!). Are those HDDs or SSDs/NVMe? >>>>> >>>>> >>>> These are SAS disks. But there is also a RAID controller with 1GB cache. >>> >>> >>>>>> When I mount the gluster vo
Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster
I don't see any other bottleneck. CPUs are quite idle. Seems that the load is mostly due to high latency on IO. Reading further the gluster docs: https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md I see that I am missing the following options: /etc/glusterfs/glusterd.vol : option rpc-auth-allow-insecure on gluster volume set vms gluster server.allow-insecure on It says that the above allow qemu to use libgfapi. When checking the VM XML, I don't see any gluster protocol at the disk drive: 222a1312-5efa-4731-8914-9a9d24dccba5 While at gluster docs it mentions the below type of disk: Does the above indicate that ovirt/qemu is not using libgfapi but FUSE only? This could be the reason of such slow perf. On Thu, Sep 7, 2017 at 1:47 PM, Yaniv Kaul wrote: > > > On Thu, Sep 7, 2017 at 12:52 PM, Abi Askushi > wrote: > >> >> >> On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul wrote: >> >>> >>> >>> On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul wrote: >>> >>>> >>>> >>>> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi >>>> wrote: >>>> >>>>> For a first idea I use: >>>>> >>>>> dd if=/dev/zero of=testfile bs=1GB count=1 >>>>> >>>> >>>> This is an incorrect way to test performance, for various reasons: >>>> 1. You are not using oflag=direct , thus not using DirectIO, but using >>>> cache. >>>> 2. It's unrealistic - it is very uncommon to write large blocks of >>>> zeros (sometimes during FS creation or wiping). Certainly not 1GB >>>> 3. It is a single thread of IO - again, unrealistic for VM's IO. >>>> >>>> I forgot to mention that I include oflag=direct in my tests. I agree >> though that dd is not the correct way to test, hence I mentioned I just use >> it to get a first feel. More tests are done within the VM benchmarking its >> disk IO (with tools like IOmeter). >> >> I suggest using fio and such. See https://github.com/pcuzner/fio-tools >>>> for example. >>>> >>> Do you have any recommended config file to use for VM workload? >> > > Desktops and Servers VMs behave quite differently, so not really. But the > 70/30 job is typically a good baseline. > > >> >> >>>> >>>>> >>>>> When testing on the gluster mount point using above command I hardly >>>>> get 10MB/s. (On the same time the network traffic hardly reaches 100Mbit). >>>>> >>>>> When testing our of the gluster (for example at /root) I get 600 - >>>>> 700MB/s. >>>>> >>>> >>>> That's very fast - from 4 disks doing RAID5? Impressive (unless you use >>>> caching!). Are those HDDs or SSDs/NVMe? >>>> >>>> >>> These are SAS disks. But there is also a RAID controller with 1GB cache. >> >> >>>>> When I mount the gluster volume with NFS and test on it I get 90 - 100 >>>>> MB/s, (almost 10x from gluster results) which is the max I can get >>>>> considering I have only 1 Gbit network for the storage. >>>>> >>>>> Also, when using glusterfs the general VM performance is very poor and >>>>> disk write benchmarks show that is it at least 4 times slower then when >>>>> the >>>>> VM is hosted on the same data store when NFS mounted. >>>>> >>>>> I don't know why I hitting such a significant performance penalty, and >>>>> every possible tweak that I was able to find out there did not make any >>>>> difference on the performance. >>>>> >>>>> The hardware I am using is pretty decent for the purposes intended: >>>>> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 >>>>> TB of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data >>>>> store of ovirt where VMs are stored. >>>>> >>>> >>> I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10? >>> Same usable capacity, higher performance, same protection and faster >>> recovery, I believe. >>> >> Correction: there are 5 disks of 600GB each. The main reason going with >> RAID 5 was the capacity. With RAID 10 I can use only 4 of them and get only >> 1.1 TB usable, with R
Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster
On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul wrote: > > > On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul wrote: > >> >> >> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi >> wrote: >> >>> For a first idea I use: >>> >>> dd if=/dev/zero of=testfile bs=1GB count=1 >>> >> >> This is an incorrect way to test performance, for various reasons: >> 1. You are not using oflag=direct , thus not using DirectIO, but using >> cache. >> 2. It's unrealistic - it is very uncommon to write large blocks of zeros >> (sometimes during FS creation or wiping). Certainly not 1GB >> 3. It is a single thread of IO - again, unrealistic for VM's IO. >> >> I forgot to mention that I include oflag=direct in my tests. I agree though that dd is not the correct way to test, hence I mentioned I just use it to get a first feel. More tests are done within the VM benchmarking its disk IO (with tools like IOmeter). I suggest using fio and such. See https://github.com/pcuzner/fio-tools for >> example. >> > Do you have any recommended config file to use for VM workload? >> >>> >>> When testing on the gluster mount point using above command I hardly get >>> 10MB/s. (On the same time the network traffic hardly reaches 100Mbit). >>> >>> When testing our of the gluster (for example at /root) I get 600 - >>> 700MB/s. >>> >> >> That's very fast - from 4 disks doing RAID5? Impressive (unless you use >> caching!). Are those HDDs or SSDs/NVMe? >> >> > These are SAS disks. But there is also a RAID controller with 1GB cache. >>> When I mount the gluster volume with NFS and test on it I get 90 - 100 >>> MB/s, (almost 10x from gluster results) which is the max I can get >>> considering I have only 1 Gbit network for the storage. >>> >>> Also, when using glusterfs the general VM performance is very poor and >>> disk write benchmarks show that is it at least 4 times slower then when the >>> VM is hosted on the same data store when NFS mounted. >>> >>> I don't know why I hitting such a significant performance penalty, and >>> every possible tweak that I was able to find out there did not make any >>> difference on the performance. >>> >>> The hardware I am using is pretty decent for the purposes intended: >>> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB >>> of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data >>> store of ovirt where VMs are stored. >>> >> > I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10? > Same usable capacity, higher performance, same protection and faster > recovery, I believe. > Correction: there are 5 disks of 600GB each. The main reason going with RAID 5 was the capacity. With RAID 10 I can use only 4 of them and get only 1.1 TB usable, with RAID 5 I get 2.2 TB usable. I agree going with RAID 10 (+ one additional drive to go with 6 drives) would be better but this is what I have now. Y. > > >> You have not mentioned your NIC speeds. Please ensure all work well, with >> 10g. >> Is the network dedicated for Gluster traffic? How are they connected? >> >> > I have mentioned that I have 1 Gbit dedicated for the storage. A different network is used for this and a dedicated 1Gbit switch. The throughput has been confirmed between all nodes with iperf. I know 10Gbit would be better, but when using native gluster at ovirt the network pipe was hardly reaching 100Mbps thus the bottleneck was gluster and not the network. If I can saturate 1Gbit and I still have performance issues then I may think to go with 10Gbit. With NFS on top gluster I see traffic reaching 800Mbit when testing with dd which is much better. >>> The gluster configuration is the following: >>> >> >> Which version of Gluster are you using? >> >> > The version is 3.8.12 > >>> Volume Name: vms >>> Type: Replicate >>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x (2 + 1) = 3 >>> Transport-type: tcp >>> Bricks: >>> Brick1: gluster0:/gluster/vms/brick >>> Brick2: gluster1:/gluster/vms/brick >>> Brick3: gluster2:/gluster/vms/brick (arbiter) >>> Options Reconfigured: >>> nfs.export-volumes: on >>> nfs.disable: off >>> performance.readdir-ahead: on >>> transport.address-family: inet >>> performance.quick-read: off >>> performance.read-ahead: off >>> performance
Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster
For a first idea I use: dd if=/dev/zero of=testfile bs=1GB count=1 When testing on the gluster mount point using above command I hardly get 10MB/s. (On the same time the network traffic hardly reaches 100Mbit). When testing our of the gluster (for example at /root) I get 600 - 700MB/s. When I mount the gluster volume with NFS and test on it I get 90 - 100 MB/s, (almost 10x from gluster results) which is the max I can get considering I have only 1 Gbit network for the storage. Also, when using glusterfs the general VM performance is very poor and disk write benchmarks show that is it at least 4 times slower then when the VM is hosted on the same data store when NFS mounted. I don't know why I hitting such a significant performance penalty, and every possible tweak that I was able to find out there did not make any difference on the performance. The hardware I am using is pretty decent for the purposes intended: 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data store of ovirt where VMs are stored. The gluster configuration is the following: Volume Name: vms Type: Replicate Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: gluster0:/gluster/vms/brick Brick2: gluster1:/gluster/vms/brick Brick3: gluster2:/gluster/vms/brick (arbiter) Options Reconfigured: nfs.export-volumes: on nfs.disable: off performance.readdir-ahead: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: on performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: off cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 1 features.shard: on user.cifs: off storage.owner-uid: 36 storage.owner-gid: 36 network.ping-timeout: 30 performance.strict-o-direct: on cluster.granular-entry-heal: enable features.shard-block-size: 64MB performance.client-io-threads: on client.event-threads: 4 server.event-threads: 4 performance.write-behind-window-size: 4MB performance.cache-size: 1GB In case I can provide any other details let me know. At the moment I already switched to gluster based NFS but I have a similar setup with 2 nodes where the data store is mounted through gluster (and again relatively good hardware) where I might check any tweaks or improvements on this setup. Thanx On Wed, Sep 6, 2017 at 5:32 PM, Yaniv Kaul wrote: > > > On Wed, Sep 6, 2017 at 3:32 PM, Abi Askushi > wrote: > >> Hi All, >> >> I've playing with ovirt self hosted engine setup and I even use it to >> production for several VM. The setup I have is 3 server with gluster >> storage in replica 2+1 (1 arbiter). >> The data storage domain where VMs are stored is mounted with gluster >> through ovirt. The performance I get for the VMs is very low and I was >> thinking to switch and mount the same storage through NFS instead of >> glusterfs. >> > > I don't see how it'll improve performance. > I suggest you share the gluster configuration (as well as the storage HW) > so we can understand why the performance is low. > Y. > > >> >> The only think I am hesitant is how can I ensure high availability of the >> storage when I loose one server? I was thinking to have at /etc/hosts sth >> like below: >> >> 10.100.100.1 nfsmount >> 10.100.100.2 nfsmount >> 10.100.100.3 nfsmount >> >> then use nfsmount as the server name when adding this domain through >> ovirt GUI. >> Are there any other more elegant solutions? What do you do for such cases? >> Note: gluster has the back-vol-file option which provides a lean way to >> have redundancy on the mount point and I am using this when mounting with >> glusterfs. >> >> Thanx >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] oVirt selft-hosted with NFS on top gluster
Hi All, I've playing with ovirt self hosted engine setup and I even use it to production for several VM. The setup I have is 3 server with gluster storage in replica 2+1 (1 arbiter). The data storage domain where VMs are stored is mounted with gluster through ovirt. The performance I get for the VMs is very low and I was thinking to switch and mount the same storage through NFS instead of glusterfs. The only think I am hesitant is how can I ensure high availability of the storage when I loose one server? I was thinking to have at /etc/hosts sth like below: 10.100.100.1 nfsmount 10.100.100.2 nfsmount 10.100.100.3 nfsmount then use nfsmount as the server name when adding this domain through ovirt GUI. Are there any other more elegant solutions? What do you do for such cases? Note: gluster has the back-vol-file option which provides a lean way to have redundancy on the mount point and I am using this when mounting with glusterfs. Thanx ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Package dependency issue
I confirm that issue has been resolved now. On Thu, Aug 24, 2017 at 12:34 PM, Abi Askushi wrote: > Thanx! > > On Thu, Aug 24, 2017 at 11:41 AM, Yedidyah Bar David > wrote: > >> On Thu, Aug 24, 2017 at 10:53 AM, Abi Askushi >> wrote: >> > Hi all, >> > >> > I have ovirt 4.1 with self hosted engine. >> > I have performed lately yum update -y on the hosts and at next yum >> update at >> > both hosts I get the following dependency issue: >> > >> > Error: Package: rubygem-fluent-plugin-secure-forward-0.4.5-1.el7.noarch >> > (centos-opstools-release) >> >Requires: rubygem(proxifier) >> > >> > The rubygem-proxifier dependency is already installed: >> > >> > rubygem-proxifier-1.0.3-1.el7.noarch >> > >> > I have tried yum update --skip-broken and rpm -Va --nofiles --nodigest >> to >> > see if this resolves this but the issue remains. >> > >> > Any idea how to resolve this? >> >> The missing package has been tagged for release and will be on CentOS >> mirrors in a few hours. >> >> See also: https://bugzilla.redhat.com/show_bug.cgi?id=1484534 >> >> Best, >> >> > >> > Thanx, >> > Abi >> > >> > ___ >> > Users mailing list >> > Users@ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > >> >> >> >> -- >> Didi >> > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Package dependency issue
Thanx! On Thu, Aug 24, 2017 at 11:41 AM, Yedidyah Bar David wrote: > On Thu, Aug 24, 2017 at 10:53 AM, Abi Askushi > wrote: > > Hi all, > > > > I have ovirt 4.1 with self hosted engine. > > I have performed lately yum update -y on the hosts and at next yum > update at > > both hosts I get the following dependency issue: > > > > Error: Package: rubygem-fluent-plugin-secure-forward-0.4.5-1.el7.noarch > > (centos-opstools-release) > >Requires: rubygem(proxifier) > > > > The rubygem-proxifier dependency is already installed: > > > > rubygem-proxifier-1.0.3-1.el7.noarch > > > > I have tried yum update --skip-broken and rpm -Va --nofiles --nodigest to > > see if this resolves this but the issue remains. > > > > Any idea how to resolve this? > > The missing package has been tagged for release and will be on CentOS > mirrors in a few hours. > > See also: https://bugzilla.redhat.com/show_bug.cgi?id=1484534 > > Best, > > > > > Thanx, > > Abi > > > > ___ > > Users mailing list > > Users@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > -- > Didi > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Package dependency issue
Hi all, I have ovirt 4.1 with self hosted engine. I have performed lately yum update -y on the hosts and at next yum update at both hosts I get the following dependency issue: Error: Package: rubygem-fluent-plugin-secure-forward-0.4.5-1.el7.noarch (centos-opstools-release) Requires: rubygem(proxifier) The rubygem-proxifier dependency is already installed: rubygem-proxifier-1.0.3-1.el7.noarch I have tried yum update --skip-broken and rpm -Va --nofiles --nodigest to see if this resolves this but the issue remains. Any idea how to resolve this? Thanx, Abi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt VM backups
Hi, Yes, the python script is using this API. I was thinking to have in the GUI for VM backups sth similar as the gluster volume snapshot schedules. On Thu, Jul 27, 2017 at 2:09 PM, Maor Lipchuk wrote: > On Thu, Jul 27, 2017 at 1:14 PM, Abi Askushi > wrote: > > Hi All, > > > > For VM backups I am using some python script to automate the snapshot -> > > clone -> export -> delete steps (although with some issues when trying to > > backups a Windows 10 VM) > > > > I was wondering if there is there any plan to integrate VM backups in the > > GUI or what other recommended ways exist out there. > > Hi Abi, > > Don't you want to use the backup API feature for that: > https://www.ovirt.org/develop/api/design/backup-api/ > > Regards, > Maor > > > > > Thanx, > > Abi > > > > ___ > > Users mailing list > > Users@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] oVirt VM backups
Hi All, For VM backups I am using some python script to automate the snapshot -> clone -> export -> delete steps (although with some issues when trying to backups a Windows 10 VM) I was wondering if there is there any plan to integrate VM backups in the GUI or what other recommended ways exist out there. Thanx, Abi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] test email
several days without receiving any email from this list. please test back. Abi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Windows USB Redirection
Hi Michal, Thank you for your tip. Do you have any example how this attribute can be extended to avoid changing the default? Thanx Abi On Wed, Jul 12, 2017 at 3:34 PM, Michal Skrivanek < michal.skriva...@redhat.com> wrote: > > On 11 Jul 2017, at 10:09, Abi Askushi wrote: > > Hi All, > > Finally I was able to pin-point the issue. > Just in case ones needs this. > > > ovirt by default enables USB 1.0 controller. > > To enable USB 2.0 controller go to > > At ovirt engine at config file below, change the controller from piix3-uhci > to piix4-uhci as below: > > > piix4-uhci is still USB1 controller > the one we use is still the most compatible default, we do not test any > other (other than set of USB2 and USB1 companions when using SPICE) > you’re welcome to overwrite it (not by changing the defaults which gets > overwritten on upgrade, but rather extending it by your custom properties, > ideally per OS type where you need that change) > You can use all the QEMU supported controllers, e.g. xhci if you want USB3 > > Thanks, > michal > > > /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties > > > os.other.devices.usb.controller.value = piix4-uhci > > > then restart engine: > systemctl restart ovirt-engine > > > USB device then is attached successfully at Windows VM. > > On Wed, Jun 28, 2017 at 12:44 PM, Abi Askushi > wrote: > >> >> Hi All, >> >> I have Ovirt 4.1 with 3 nodes on top glusterfs. >> >> I Have 2 VMs: Windows 2016 64bit and Windows 10 64bit >> >> When I attach a USB flash disk to a VM (from host devices) the VM cannot >> see the USB drive and report driver issue at device manager (see attached). >> This happens with both VMs when tested. >> >> When testing with Windows 7 or Windows XP USB is attached and accessed >> normally. >> Have you encountered such issue? >> I have installed latest guest tools on both VMs. >> >> Many thanx >> >> > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Windows USB Redirection
Hi All, Finally I was able to pin-point the issue. Just in case ones needs this. ovirt by default enables USB 1.0 controller. To enable USB 2.0 controller go to At ovirt engine at config file below, change the controller from piix3-uhci to piix4-uhci as below: /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties os.other.devices.usb.controller.value = piix4-uhci then restart engine: systemctl restart ovirt-engine USB device then is attached successfully at Windows VM. On Wed, Jun 28, 2017 at 12:44 PM, Abi Askushi wrote: > > Hi All, > > I have Ovirt 4.1 with 3 nodes on top glusterfs. > > I Have 2 VMs: Windows 2016 64bit and Windows 10 64bit > > When I attach a USB flash disk to a VM (from host devices) the VM cannot > see the USB drive and report driver issue at device manager (see attached). > This happens with both VMs when tested. > > When testing with Windows 7 or Windows XP USB is attached and accessed > normally. > Have you encountered such issue? > I have installed latest guest tools on both VMs. > > Many thanx > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Fwd: Windows USB Redirection
Hi All, I have Ovirt 4.1 with 3 nodes on top glusterfs. I Have 2 VMs: Windows 2016 64bit and Windows 10 64bit When I attach a USB flash disk to a VM (from host devices) the VM cannot see the USB drive and report driver issue at device manager (see attached). This happens with both VMs when tested. When testing with Windows 7 or Windows XP USB is attached and accessed normally. Have you encountered such issue? I have installed latest guest tools on both VMs. Many thanx ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Ovirt remove network from hosted engine
Hi, I was trying to remove the nic from the edit menu of the VM and that failed. I had to unplug the nic first to be able to remove it from the network interfaces submenu of the vm. On Jun 28, 2017 08:29, "knarra" wrote: > On 06/27/2017 09:49 PM, Abi Askushi wrote: > > Hi all, > > Just in case ones needs it, in order to remove the secondary network > interface from engine, you can go to: > Virtual Machines -> Hostedengine -> Network Interfaces -> edit -> unplug > it -> confirm -> remove it. > > cool. But in your previous mail you did mention that it fails for you > since the engine is running. Instead of remove you tried unplug here ? > > > It was simple... > > > On Tue, Jun 27, 2017 at 4:54 PM, Abi Askushi > wrote: > >> Hi Knarra, >> >> Then I had already enabled NFS on ISO gluster volume. >> Maybe i had some networking issue then. I need to remove the secondary >> interface in order to test that again. >> >> >> >> On Tue, Jun 27, 2017 at 4:25 PM, knarra wrote: >> >>> On 06/27/2017 06:34 PM, Abi Askushi wrote: >>> >>> Hi Knarra, >>> >>> The ISO domain is of type gluster though I had nfs enabled on that >>> volume. >>> >>> you need to have nfs enabled on the volume. what i meant is nfs.disable >>> off which means nfs is on. >>> >>> For more info please refer to bug https://bugzilla.redhat.com/sh >>> ow_bug.cgi?id=1437799 >>> >>> I will disable the nfs and try. Though in order to try I need first to >>> remove that second interface from engine. >>> Is there a way I can remove the secondary storage network interface from >>> the engine? >>> >>> I am not sure how to do that, but you may shutdown the vm using the >>> command hosted-engine --vm-shutdown which will power off the vm and try to >>> remove the networks using vdsclient. (not sure if this is right, but >>> suggesting a way) >>> >>> >>> Thanx >>> >>> >>> >>> On Tue, Jun 27, 2017 at 3:32 PM, knarra wrote: >>> >>>> On 06/27/2017 05:41 PM, Abi Askushi wrote: >>>> >>>> Hi all, >>>> >>>> When setting up hosted engine setup on top gluster with 3 nodes, I had >>>> gluster configured on a separate network interface, as recommended. When I >>>> tried later to upload ISO from engine to ISO domain, the engine was not >>>> able to upload it since the VM did not have access to the separate storage >>>> network. I then added the storage network interface to the hosted engine >>>> and ISO upload succeeded. >>>> >>>> May i know what was the volume type created and added as ISO domain ? >>>> >>>> If you plan to use a glusterfs volume below is the procedure : >>>> >>>> 1) Create a glusterfs volume. >>>> 2) While adding storage domain select Domain Function as 'ISO' and >>>> Storage Type as 'glusterfs' . >>>> 3) You can either use 'use managed gluster volume' check box and select >>>> the gluster volume which you have created for storing ISO's or you can type >>>> the full path of the volume. >>>> 4) Once this is added please make sure to set the option nfs.disable >>>> off. >>>> 5) Now you can go to HE engine and run the command engine-iso-uploader >>>> upload -i >>>> >>>> Iso gets uploaded successfully. >>>> >>>> >>>> 1st question: do I need to add the network interface to engine in order >>>> to upload ISOs? does there exist any alternate way? >>>> >>>> AFAIK, this is not required when glusterfs volume is used. >>>> >>>> Attached is the screenshot where i have only one network attached to my >>>> HE which is ovirtmgmt. >>>> >>>> >>>> Then I proceeded to configure bonding for the storage domain, bonding 2 >>>> NICs at each server. When trying to set a custom bond of mode=6 (as >>>> recommended from gluster) I received a warning that mode0, 5 and 6 cannot >>>> be configured since the interface is used from VMs. I also understood that >>>> having the storage network assigned to VMs makes it a bridge which >>>> decreases performance of networking. When trying to remove the network >>>> interface from engine it cannot be done, since the engine is running. >>>> >>>> 2nd question: Is there a way I can remove the secondary storage network >>>> interface from the engine? >>>> >>>> Many thanx >>>> >>>> >>>> ___ >>>> Users mailing >>>> listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >>>> >>>> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Ovirt remove network from hosted engine
Hi all, Just in case ones needs it, in order to remove the secondary network interface from engine, you can go to: Virtual Machines -> Hostedengine -> Network Interfaces -> edit -> unplug it -> confirm -> remove it. It was simple... On Tue, Jun 27, 2017 at 4:54 PM, Abi Askushi wrote: > Hi Knarra, > > Then I had already enabled NFS on ISO gluster volume. > Maybe i had some networking issue then. I need to remove the secondary > interface in order to test that again. > > > > On Tue, Jun 27, 2017 at 4:25 PM, knarra wrote: > >> On 06/27/2017 06:34 PM, Abi Askushi wrote: >> >> Hi Knarra, >> >> The ISO domain is of type gluster though I had nfs enabled on that >> volume. >> >> you need to have nfs enabled on the volume. what i meant is nfs.disable >> off which means nfs is on. >> >> For more info please refer to bug https://bugzilla.redhat.com/sh >> ow_bug.cgi?id=1437799 >> >> I will disable the nfs and try. Though in order to try I need first to >> remove that second interface from engine. >> Is there a way I can remove the secondary storage network interface from >> the engine? >> >> I am not sure how to do that, but you may shutdown the vm using the >> command hosted-engine --vm-shutdown which will power off the vm and try to >> remove the networks using vdsclient. (not sure if this is right, but >> suggesting a way) >> >> >> Thanx >> >> >> >> On Tue, Jun 27, 2017 at 3:32 PM, knarra wrote: >> >>> On 06/27/2017 05:41 PM, Abi Askushi wrote: >>> >>> Hi all, >>> >>> When setting up hosted engine setup on top gluster with 3 nodes, I had >>> gluster configured on a separate network interface, as recommended. When I >>> tried later to upload ISO from engine to ISO domain, the engine was not >>> able to upload it since the VM did not have access to the separate storage >>> network. I then added the storage network interface to the hosted engine >>> and ISO upload succeeded. >>> >>> May i know what was the volume type created and added as ISO domain ? >>> >>> If you plan to use a glusterfs volume below is the procedure : >>> >>> 1) Create a glusterfs volume. >>> 2) While adding storage domain select Domain Function as 'ISO' and >>> Storage Type as 'glusterfs' . >>> 3) You can either use 'use managed gluster volume' check box and select >>> the gluster volume which you have created for storing ISO's or you can type >>> the full path of the volume. >>> 4) Once this is added please make sure to set the option nfs.disable off. >>> 5) Now you can go to HE engine and run the command engine-iso-uploader >>> upload -i >>> >>> Iso gets uploaded successfully. >>> >>> >>> 1st question: do I need to add the network interface to engine in order >>> to upload ISOs? does there exist any alternate way? >>> >>> AFAIK, this is not required when glusterfs volume is used. >>> >>> Attached is the screenshot where i have only one network attached to my >>> HE which is ovirtmgmt. >>> >>> >>> Then I proceeded to configure bonding for the storage domain, bonding 2 >>> NICs at each server. When trying to set a custom bond of mode=6 (as >>> recommended from gluster) I received a warning that mode0, 5 and 6 cannot >>> be configured since the interface is used from VMs. I also understood that >>> having the storage network assigned to VMs makes it a bridge which >>> decreases performance of networking. When trying to remove the network >>> interface from engine it cannot be done, since the engine is running. >>> >>> 2nd question: Is there a way I can remove the secondary storage network >>> interface from the engine? >>> >>> Many thanx >>> >>> >>> ___ >>> Users mailing >>> listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >>> >>> > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Ovirt remove network from hosted engine
Hi Knarra, Then I had already enabled NFS on ISO gluster volume. Maybe i had some networking issue then. I need to remove the secondary interface in order to test that again. On Tue, Jun 27, 2017 at 4:25 PM, knarra wrote: > On 06/27/2017 06:34 PM, Abi Askushi wrote: > > Hi Knarra, > > The ISO domain is of type gluster though I had nfs enabled on that volume. > > you need to have nfs enabled on the volume. what i meant is nfs.disable > off which means nfs is on. > > For more info please refer to bug https://bugzilla.redhat.com/ > show_bug.cgi?id=1437799 > > I will disable the nfs and try. Though in order to try I need first to > remove that second interface from engine. > Is there a way I can remove the secondary storage network interface from > the engine? > > I am not sure how to do that, but you may shutdown the vm using the > command hosted-engine --vm-shutdown which will power off the vm and try to > remove the networks using vdsclient. (not sure if this is right, but > suggesting a way) > > > Thanx > > > > On Tue, Jun 27, 2017 at 3:32 PM, knarra wrote: > >> On 06/27/2017 05:41 PM, Abi Askushi wrote: >> >> Hi all, >> >> When setting up hosted engine setup on top gluster with 3 nodes, I had >> gluster configured on a separate network interface, as recommended. When I >> tried later to upload ISO from engine to ISO domain, the engine was not >> able to upload it since the VM did not have access to the separate storage >> network. I then added the storage network interface to the hosted engine >> and ISO upload succeeded. >> >> May i know what was the volume type created and added as ISO domain ? >> >> If you plan to use a glusterfs volume below is the procedure : >> >> 1) Create a glusterfs volume. >> 2) While adding storage domain select Domain Function as 'ISO' and >> Storage Type as 'glusterfs' . >> 3) You can either use 'use managed gluster volume' check box and select >> the gluster volume which you have created for storing ISO's or you can type >> the full path of the volume. >> 4) Once this is added please make sure to set the option nfs.disable off. >> 5) Now you can go to HE engine and run the command engine-iso-uploader >> upload -i >> >> Iso gets uploaded successfully. >> >> >> 1st question: do I need to add the network interface to engine in order >> to upload ISOs? does there exist any alternate way? >> >> AFAIK, this is not required when glusterfs volume is used. >> >> Attached is the screenshot where i have only one network attached to my >> HE which is ovirtmgmt. >> >> >> Then I proceeded to configure bonding for the storage domain, bonding 2 >> NICs at each server. When trying to set a custom bond of mode=6 (as >> recommended from gluster) I received a warning that mode0, 5 and 6 cannot >> be configured since the interface is used from VMs. I also understood that >> having the storage network assigned to VMs makes it a bridge which >> decreases performance of networking. When trying to remove the network >> interface from engine it cannot be done, since the engine is running. >> >> 2nd question: Is there a way I can remove the secondary storage network >> interface from the engine? >> >> Many thanx >> >> >> ___ >> Users mailing >> listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >> >> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Ovirt remove network from hosted engine
Hi Knarra, The ISO domain is of type gluster though I had nfs enabled on that volume. I will disable the nfs and try. Though in order to try I need first to remove that second interface from engine. Is there a way I can remove the secondary storage network interface from the engine? Thanx On Tue, Jun 27, 2017 at 3:32 PM, knarra wrote: > On 06/27/2017 05:41 PM, Abi Askushi wrote: > > Hi all, > > When setting up hosted engine setup on top gluster with 3 nodes, I had > gluster configured on a separate network interface, as recommended. When I > tried later to upload ISO from engine to ISO domain, the engine was not > able to upload it since the VM did not have access to the separate storage > network. I then added the storage network interface to the hosted engine > and ISO upload succeeded. > > May i know what was the volume type created and added as ISO domain ? > > If you plan to use a glusterfs volume below is the procedure : > > 1) Create a glusterfs volume. > 2) While adding storage domain select Domain Function as 'ISO' and Storage > Type as 'glusterfs' . > 3) You can either use 'use managed gluster volume' check box and select > the gluster volume which you have created for storing ISO's or you can type > the full path of the volume. > 4) Once this is added please make sure to set the option nfs.disable off. > 5) Now you can go to HE engine and run the command engine-iso-uploader > upload -i > > Iso gets uploaded successfully. > > > 1st question: do I need to add the network interface to engine in order to > upload ISOs? does there exist any alternate way? > > AFAIK, this is not required when glusterfs volume is used. > > Attached is the screenshot where i have only one network attached to my HE > which is ovirtmgmt. > > > Then I proceeded to configure bonding for the storage domain, bonding 2 > NICs at each server. When trying to set a custom bond of mode=6 (as > recommended from gluster) I received a warning that mode0, 5 and 6 cannot > be configured since the interface is used from VMs. I also understood that > having the storage network assigned to VMs makes it a bridge which > decreases performance of networking. When trying to remove the network > interface from engine it cannot be done, since the engine is running. > > 2nd question: Is there a way I can remove the secondary storage network > interface from the engine? > > Many thanx > > > ___ > Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users > > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Ovirt remove network from hosted engine
Hi all, When setting up hosted engine setup on top gluster with 3 nodes, I had gluster configured on a separate network interface, as recommended. When I tried later to upload ISO from engine to ISO domain, the engine was not able to upload it since the VM did not have access to the separate storage network. I then added the storage network interface to the hosted engine and ISO upload succeeded. 1st question: do I need to add the network interface to engine in order to upload ISOs? does there exist any alternate way? Then I proceeded to configure bonding for the storage domain, bonding 2 NICs at each server. When trying to set a custom bond of mode=6 (as recommended from gluster) I received a warning that mode0, 5 and 6 cannot be configured since the interface is used from VMs. I also understood that having the storage network assigned to VMs makes it a bridge which decreases performance of networking. When trying to remove the network interface from engine it cannot be done, since the engine is running. 2nd question: Is there a way I can remove the secondary storage network interface from the engine? Many thanx ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt split brain resolution
Hi Satheesaran, gluster volume info engine Volume Name: engine Type: Replicate Volume ID: 3caae601-74dd-40d1-8629-9a61072bec0f Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: gluster0:/gluster/engine/brick Brick2: gluster1:/gluster/engine/brick Brick3: gluster2:/gluster/engine/brick (arbiter) Options Reconfigured: nfs.disable: on performance.readdir-ahead: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 1 features.shard: on user.cifs: off storage.owner-uid: 36 storage.owner-gid: 36 network.ping-timeout: 30 performance.strict-o-direct: on cluster.granular-entry-heal: enable nfs.export-volumes: on As per my previous, i have resolved this by following the steps described. On Tue, Jun 27, 2017 at 1:42 PM, Satheesaran Sundaramoorthi < sasun...@redhat.com> wrote: > On Sat, Jun 24, 2017 at 3:17 PM, Abi Askushi > wrote: > >> Hi all, >> >> For the records, I had to remove manually the conflicting directory and >> ts respective gfid from the arbiter volume: >> >> getfattr -m . -d -e hex e1c80750-b880-495e-9609-b8bc7760d101/ha_agent >> >> That gave me the gfid: 0x277c9caa9dce4a17a2a93775357befd5 >> >> Then cd .glusterfs/27/7c >> >> rm -rf 277c9caa-9dce-4a17-a2a9-3775357befd5 (or move it out of there) >> >> Triggerred heal: gluster volume heal engine >> >> Then all ok: >> >> gluster volume heal engine info >> Brick gluster0:/gluster/engine/brick >> Status: Connected >> Number of entries: 0 >> >> Brick gluster1:/gluster/engine/brick >> Status: Connected >> Number of entries: 0 >> >> Brick gluster2:/gluster/engine/brick >> Status: Connected >> Number of entries: 0 >> >> Thanx. >> > > Hi Abi, > > What is the volume type of 'engine' volume ? > Could you also provide the output of 'gluster volume info engine' to get > to the > closer look at the problem > > -- sas > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt split brain resolution
Hi all,
For the records, I had to remove manually the conflicting directory and ts
respective gfid from the arbiter volume:
getfattr -m . -d -e hex e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
That gave me the gfid: 0x277c9caa9dce4a17a2a93775357befd5
Then cd .glusterfs/27/7c
rm -rf 277c9caa-9dce-4a17-a2a9-3775357befd5 (or move it out of there)
Triggerred heal: gluster volume heal engine
Then all ok:
gluster volume heal engine info
Brick gluster0:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick gluster1:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick gluster2:/gluster/engine/brick
Status: Connected
Number of entries: 0
Thanx.
On Fri, Jun 23, 2017 at 7:21 PM, Abi Askushi
wrote:
> Hi Denis,
>
> I receive permission denied as below:
>
> gluster volume heal engine split-brain latest-mtime
> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
> Healing /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent failed:Operation
> not permitted.
> Volume heal failed.
>
>
> When I shutdown host3 then no split brain is reported from the remaining
> two hosts. When I power up host3 then I receive the mentioned split brain
> and host3 logs the following at ovirt-hosted-engine-ha/agent.log
>
> MainThread::INFO::2017-06-23 16:18:06,067::hosted_engine::
> 594::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_broker) Failed set the storage domain: 'Failed
> to set storage domain VdsmBackend, options {'hosted-engine.lockspace': '
> 7B22696D6167655F75756964223A202238323132626637382D66392D
> 346465652D61672D346265633734353035366235222C202270617468
> 223A206E756C6C2C2022766F6C756D655F75756964223A20223632373930
> 3162652D666261332D346263342D393037632D393931356138333632633537227D',
> 'sp_uuid': '----', 'dom_type':
> 'glusterfs', 'hosted-engine.metadata': '7B22696D6167655F75756964223A20
> 2263353930633034372D613462322D346539312D613832362D6434386239
> 61643537323330222C202270617468223A206E756C6C2C2022766F6C756D
> 655F75756964223A202230353166653865612D39632D346134302D38
> 3438382D386335313138666438373238227D', 'sd_uuid':
> 'e1c80750-b880-495e-9609-b8bc7760d101'}: Request failed: 'exceptions.OSError'>'. Waiting '5's before the next attempt
>
> and the following at /var/log/messages:
> Jun 23 16:19:43 v2 journal: vdsm root ERROR failed to retrieve Hosted
> Engine HA info#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
> _getHaInfo#012stats = instance.get_all_stats()#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 105, in get_all_stats#012stats =
> broker.get_stats_from_storage(service)#012
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 233, in get_stats_from_storage#012result =
> self._checked_communicate(request)#012 File "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 261, in
> _checked_communicate#012.format(message or response))#012RequestError:
> Request failed: failed to read metadata: [Errno 5] Input/output error:
> '/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/e1c80750-b880-495e-
> 9609-b8bc7760d101/ha_agent/hosted-engine.metadata'
>
> Thanx
>
>
> On Fri, Jun 23, 2017 at 6:05 PM, Denis Chaplygin
> wrote:
>
>> Hello Abi,
>>
>> On Fri, Jun 23, 2017 at 4:47 PM, Abi Askushi
>> wrote:
>>
>>> Hi All,
>>>
>>> I have a 3 node ovirt 4.1 setup. I lost one node due to raid controller
>>> issues. Upon restoration I have the following split brain, although the
>>> hosts have mounted the storage domains:
>>>
>>> gluster volume heal engine info split-brain
>>> Brick gluster0:/gluster/engine/brick
>>> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
>>> Status: Connected
>>> Number of entries in split-brain: 1
>>>
>>> Brick gluster1:/gluster/engine/brick
>>> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
>>> Status: Connected
>>> Number of entries in split-brain: 1
>>>
>>> Brick gluster2:/gluster/engine/brick
>>> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
>>> Status: Connected
>>> Number of entries in split-brain: 1
>>>
>>>
>>>
>> It is definitely on gluster side. You could try to use
>>
>> gluster volume heal engine split-brain latest-mtime /e1c80750-b880-49
>> 5e-9609-b8bc7760d101/ha_agent
>>
>>
>> I also added gluster developers to that thread, so they may provide you
>> with better advices.
>>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt split brain resolution
Hi Denis,
I receive permission denied as below:
gluster volume heal engine split-brain latest-mtime
/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
Healing /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent failed:Operation not
permitted.
Volume heal failed.
When I shutdown host3 then no split brain is reported from the remaining
two hosts. When I power up host3 then I receive the mentioned split brain
and host3 logs the following at ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2017-06-23
16:18:06,067::hosted_engine::594::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
Failed set the storage domain: 'Failed to set storage domain VdsmBackend,
options {'hosted-engine.lockspace':
'7B22696D6167655F75756964223A202238323132626637382D66392D346465652D61672D346265633734353035366235222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A202236323739303162652D666261332D346263342D393037632D393931356138333632633537227D',
'sp_uuid': '----', 'dom_type': 'glusterfs',
'hosted-engine.metadata':
'7B22696D6167655F75756964223A202263353930633034372D613462322D346539312D613832362D643438623961643537323330222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A202230353166653865612D39632D346134302D383438382D386335313138666438373238227D',
'sd_uuid': 'e1c80750-b880-495e-9609-b8bc7760d101'}: Request failed: '. Waiting '5's before the next attempt
and the following at /var/log/messages:
Jun 23 16:19:43 v2 journal: vdsm root ERROR failed to retrieve Hosted
Engine HA info#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
_getHaInfo#012stats = instance.get_all_stats()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 105, in get_all_stats#012stats =
broker.get_stats_from_storage(service)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 233, in get_stats_from_storage#012result =
self._checked_communicate(request)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 261, in _checked_communicate#012.format(message or
response))#012RequestError: Request failed: failed to read metadata: [Errno
5] Input/output error: '/rhev/data-center/mnt/glusterSD/10.100.100.1:
_engine/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent/hosted-engine.metadata'
Thanx
On Fri, Jun 23, 2017 at 6:05 PM, Denis Chaplygin
wrote:
> Hello Abi,
>
> On Fri, Jun 23, 2017 at 4:47 PM, Abi Askushi
> wrote:
>
>> Hi All,
>>
>> I have a 3 node ovirt 4.1 setup. I lost one node due to raid controller
>> issues. Upon restoration I have the following split brain, although the
>> hosts have mounted the storage domains:
>>
>> gluster volume heal engine info split-brain
>> Brick gluster0:/gluster/engine/brick
>> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
>> Status: Connected
>> Number of entries in split-brain: 1
>>
>> Brick gluster1:/gluster/engine/brick
>> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
>> Status: Connected
>> Number of entries in split-brain: 1
>>
>> Brick gluster2:/gluster/engine/brick
>> /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
>> Status: Connected
>> Number of entries in split-brain: 1
>>
>>
>>
> It is definitely on gluster side. You could try to use
>
> gluster volume heal engine split-brain latest-mtime /e1c80750-b880-
> 495e-9609-b8bc7760d101/ha_agent
>
>
> I also added gluster developers to that thread, so they may provide you
> with better advices.
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] oVirt split brain resolution
Hi All, I have a 3 node ovirt 4.1 setup. I lost one node due to raid controller issues. Upon restoration I have the following split brain, although the hosts have mounted the storage domains: gluster volume heal engine info split-brain Brick gluster0:/gluster/engine/brick /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent Status: Connected Number of entries in split-brain: 1 Brick gluster1:/gluster/engine/brick /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent Status: Connected Number of entries in split-brain: 1 Brick gluster2:/gluster/engine/brick /e1c80750-b880-495e-9609-b8bc7760d101/ha_agent Status: Connected Number of entries in split-brain: 1 Hosted engine status gives the following: hosted-engine --vm-status Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 173, in if not status_checker.print_status(): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 103, in print_status all_host_stats = self._get_all_host_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 73, in _get_all_host_stats all_host_stats = ha_cli.get_all_host_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 160, in get_all_host_stats return self.get_all_stats(self.StatModes.HOST) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 105, in get_all_stats stats = broker.get_stats_from_storage(service) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 233, in get_stats_from_storage result = self._checked_communicate(request) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 261, in _checked_communicate .format(message or response)) ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed to read metadata: [Errno 5] Input/output error: '/rhev/data-center/mnt/glusterSD/10.100.100.1: _engine/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent/hosted-engine.metadata' Any idea on how to resolve this split brain? Thanx, Alex ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Win server 2016 guest tools
Thank you Lev. I confirm that these tools install successfully. Alex On Tue, Jun 20, 2017 at 4:04 PM, Lev Veyde wrote: > Hi, > > You can use the tools from: http://plain.resources.ovirt. > org/pub/ovirt-4.1/iso/oVirt-toolsSetup/oVirt-toolsSetup-4.1-5.fc24.iso > > It should support MS Windows Server 2016. > > Thanks in advance, > > On Tue, Jun 20, 2017 at 11:26 AM, Abi Askushi > wrote: > >> Hi all, >> >> Are there any windows server 2016 guest tools? At the site it doesn't >> list this OS and during installation from ISO that is included in >> ovirt-guest-tools-iso package i receive the error that the OS is not >> supported. >> >> If these tools are not installed and I install only virtio tools are >> there any performance implications for the VM? >> >> Thanx, >> Alex >> >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > > -- > > Lev Veyde > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > l...@redhat.com | lve...@redhat.com > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Win server 2016 guest tools
Hi all, Are there any windows server 2016 guest tools? At the site it doesn't list this OS and during installation from ISO that is included in ovirt-guest-tools-iso package i receive the error that the OS is not supported. If these tools are not installed and I install only virtio tools are there any performance implications for the VM? Thanx, Alex ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt gluster sanlock issue
Filed the *Bug 1459855* <https://bugzilla.redhat.com/show_bug.cgi?id=1459855> Alex On Thu, Jun 8, 2017 at 1:16 PM, Abi Askushi wrote: > Hi Denis, > > Ok I will file a bug for this. > I am not sure if I will be able to provide troubleshooting info for much > long as I already have put forward the replacement of disks with 512 ones. > > Alex > > On Thu, Jun 8, 2017 at 11:48 AM, Denis Chaplygin > wrote: > >> Hello Alex, >> >> >> On Wed, Jun 7, 2017 at 11:39 AM, Abi Askushi >> wrote: >> >>> Hi Sahina, >>> >>> Did you have the chance to check the logs and have any idea how this may >>> be addressed? >>> >> >> >> It seems to be a VDSM issue, as VDSM uses direct IO (and id actualy calls >> dd) and assumes that block size is 512. I see in the code, that block size >> is defined as a constant, so it probably may be adjusted, but i think it >> would be better if we ask some one who knows that part better. >> >> Anyway, could you please file a bug on that issue? Thanks in advance. >> > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt gluster sanlock issue
Hi Denis, Ok I will file a bug for this. I am not sure if I will be able to provide troubleshooting info for much long as I already have put forward the replacement of disks with 512 ones. Alex On Thu, Jun 8, 2017 at 11:48 AM, Denis Chaplygin wrote: > Hello Alex, > > > On Wed, Jun 7, 2017 at 11:39 AM, Abi Askushi > wrote: > >> Hi Sahina, >> >> Did you have the chance to check the logs and have any idea how this may >> be addressed? >> > > > It seems to be a VDSM issue, as VDSM uses direct IO (and id actualy calls > dd) and assumes that block size is 512. I see in the code, that block size > is defined as a constant, so it probably may be adjusted, but i think it > would be better if we ask some one who knows that part better. > > Anyway, could you please file a bug on that issue? Thanks in advance. > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] oVirt gluster sanlock issue
Hi Sahina, Did you have the chance to check the logs and have any idea how this may be addressed? Thanx, Alex On Mon, Jun 5, 2017 at 12:14 PM, Sahina Bose wrote: > Can we have the gluster mount logs and brick logs to check if it's the > same issue? > > On Sun, Jun 4, 2017 at 11:21 PM, Abi Askushi > wrote: > >> I clean installed everything and ran into the same. >> I then ran gdeploy and encountered the same issue when deploying engine. >> Seems that gluster (?) doesn't like 4K sector drives. I am not sure if it >> has to do with alignment. The weird thing is that gluster volumes are all >> ok, replicating normally and no split brain is reported. >> >> The solution to the mentioned bug (1386443 >> <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format >> with 512 sector size, which for my case is not an option: >> >> mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine >> illegal sector size 512; hw sector is 4096 >> >> Is there any workaround to address this? >> >> Thanx, >> Alex >> >> >> On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi >> wrote: >> >>> Hi Maor, >>> >>> My disk are of 4K block size and from this bug seems that gluster >>> replica needs 512B block size. >>> Is there a way to make gluster function with 4K drives? >>> >>> Thank you! >>> >>> On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk >>> wrote: >>> >>>> Hi Alex, >>>> >>>> I saw a bug that might be related to the issue you encountered at >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1386443 >>>> >>>> Sahina, maybe you have any advise? Do you think that BZ1386443is >>>> related? >>>> >>>> Regards, >>>> Maor >>>> >>>> On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi >>>> wrote: >>>> > Hi All, >>>> > >>>> > I have installed successfully several times oVirt (version 4.1) with >>>> 3 nodes >>>> > on top glusterfs. >>>> > >>>> > This time, when trying to configure the same setup, I am facing the >>>> > following issue which doesn't seem to go away. During installation i >>>> get the >>>> > error: >>>> > >>>> > Failed to execute stage 'Misc configuration': Cannot acquire host id: >>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>> 'Sanlock >>>> > lockspace add failure', 'Invalid argument')) >>>> > >>>> > The only different in this setup is that instead of standard >>>> partitioning i >>>> > have GPT partitioning and the disks have 4K block size instead of 512. >>>> > >>>> > The /var/log/sanlock.log has the following lines: >>>> > >>>> > 2017-06-03 19:21:15+0200 23450 [943]: s9 lockspace >>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:250:/rhev/data-center/m >>>> nt/_var_lib_ovirt-hosted-engin-setup_tmptjkIDI/ba6bd862-c2b8 >>>> -46e7-b2c8-91e4a5bb2047/dom_md/ids:0 >>>> > 2017-06-03 19:21:36+0200 23471 [944]: s9:r5 resource >>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:SDM:/rhev/data-center/m >>>> nt/_var_lib_ovirt-hosted-engine-setup_tmptjkIDI/ba6bd862-c2b >>>> 8-46e7-b2c8-91e4a5bb2047/dom_md/leases:1048576 >>>> > for 2,9,23040 >>>> > 2017-06-03 19:21:36+0200 23471 [943]: s10 lockspace >>>> > a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922:250:/rhev/data-center/m >>>> nt/glusterSD/10.100.100.1:_engine/a5a6b0e7-fc3f-4838-8e26-c8 >>>> b4d5e5e922/dom_md/ids:0 >>>> > 2017-06-03 19:21:36+0200 23471 [23522]: a5a6b0e7 aio collect RD >>>> > 0x7f59b8c0:0x7f59b8d0:0x7f59b0101000 result -22:0 match res >>>> > 2017-06-03 19:21:36+0200 23471 [23522]: read_sectors delta_leader >>>> offset >>>> > 127488 rv -22 >>>> > /rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/a5a6b0e >>>> 7-fc3f-4838-8e26-c8b4d5e5e922/dom_md/ids >>>> > 2017-06-03 19:21:37+0200 23472 [930]: s9 host 250 1 23450 >>>> > 88c2244c-a782-40ed-9560-6cfa4d46f853.v0.neptune >>>> > 2017-06-03 19:21:37+0200 23472 [943]: s10 add_lockspace fail result >>>> -22 >>>> > >>>> > And /var/log/vdsm/vdsm.log says: >>>> > >&
Re: [ovirt-users] oVirt gluster sanlock issue
Just to note that the mentioned logs below are from the dd with bs=512, which were failing. Attached the full logs from mount and brick. Alex On Tue, Jun 6, 2017 at 3:18 PM, Abi Askushi wrote: > Hi Krutika, > > My comments inline. > > Also attached the strace of: > strace -y -ff -o /root/512-trace-on-root.log dd if=/dev/zero > of=/mnt/test2.img oflag=direct bs=512 count=1 > > and of: > strace -y -ff -o /root/4096-trace-on-root.log dd if=/dev/zero > of=/mnt/test2.img oflag=direct bs=4096 count=16 > > I have mounted gluster volume at /mnt. > The dd with bs=4096 is successful. > > The gluster mount log gives only the following: > [2017-06-06 12:04:54.102576] W [MSGID: 114031] > [client-rpc-fops.c:854:client3_3_writev_cbk] > 0-engine-client-0: remote operation failed [Invalid argument] > [2017-06-06 12:04:54.102591] W [MSGID: 114031] > [client-rpc-fops.c:854:client3_3_writev_cbk] > 0-engine-client-1: remote operation failed [Invalid argument] > [2017-06-06 12:04:54.103355] W [fuse-bridge.c:2312:fuse_writev_cbk] > 0-glusterfs-fuse: 205: WRITE => -1 gfid=075ab3a5-0274-4f07-a075-2748c3b4d394 > fd=0x7faf1d08706c (Transport endpoint is not connected) > > The gluster brick log gives: > [2017-06-06 12:07:03.793080] E [MSGID: 113072] [posix.c:3453:posix_writev] > 0-engine-posix: write failed: offset 0, [Invalid argument] > [2017-06-06 12:07:03.793172] E [MSGID: 115067] > [server-rpc-fops.c:1346:server_writev_cbk] > 0-engine-server: 291: WRITEV 0 (075ab3a5-0274-4f07-a075-2748c3b4d394) ==> > (Invalid argument) [Invalid argument] > > > > On Tue, Jun 6, 2017 at 12:50 PM, Krutika Dhananjay > wrote: > >> OK. >> >> So for the 'Transport endpoint is not connected' issue, could you share >> the mount and brick logs? >> >> Hmmm.. 'Invalid argument' error even on the root partition. What if you >> change bs to 4096 and run? >> > If I use bs=4096 the dd is successful on /root and at gluster mounted > volume. > >> >> The logs I showed in my earlier mail shows that gluster is merely >> returning the error it got from the disk file system where the >> brick is hosted. But you're right about the fact that the offset 127488 >> is not 4K-aligned. >> >> If the dd on /root worked for you with bs=4096, could you try the same >> directly on gluster mount point on a dummy file and capture the strace >> output of dd? >> You can perhaps reuse your existing gluster volume by mounting it at >> another location and doing the dd. >> Here's what you need to execute: >> >> strace -ff -T -p -o >> ` >> >> FWIW, here's something I found in man(2) open: >> >> >> >> >> *Under Linux 2.4, transfer sizes, and the alignment of the user >> buffer and the file offset must all be multiples of the logical block size >> of the filesystem. Since Linux 2.6.0, alignment to the logical block size >> of the underlying storage (typically 512 bytes) suffices. The >> logical block size can be determined using the ioctl(2) BLKSSZGET operation >> or from the shell using the command: blockdev --getss* >> > Please note also that the physical disks are of 4K sector size (native). > Thus OS is having 4096/4096 local/physical sector size. > [root@v0 ~]# blockdev --getss /dev/sda > 4096 > [root@v0 ~]# blockdev --getpbsz /dev/sda > 4096 > >> >> >> -Krutika >> >> >> On Tue, Jun 6, 2017 at 1:18 AM, Abi Askushi >> wrote: >> >>> Also when testing with dd i get the following: >>> >>> *Testing on the gluster mount: * >>> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/10.100.100.1: >>> _engine/test2.img oflag=direct bs=512 count=1 >>> dd: error writing β/rhev/data-center/mnt/glusterSD/10.100.100.1: >>> _engine/test2.imgβ: *Transport endpoint is not connected* >>> 1+0 records in >>> 0+0 records out >>> 0 bytes (0 B) copied, 0.00336755 s, 0.0 kB/s >>> >>> *Testing on the /root directory (XFS): * >>> dd if=/dev/zero of=/test2.img oflag=direct bs=512 count=1 >>> dd: error writing β/test2.imgβ:* Invalid argument* >>> 1+0 records in >>> 0+0 records out >>> 0 bytes (0 B) copied, 0.000321239 s, 0.0 kB/s >>> >>> Seems that the gluster is trying to do the same and fails. >>> >>> >>> >>> On Mon, Jun 5, 2017 at 10:10 PM, Abi Askushi >>> wrote: >>> >>>> The question that rises is what is needed to make gluster aware of the >>>> 4K physical sectors presented to it (the l
Re: [ovirt-users] oVirt gluster sanlock issue
Also I see this article from Redhat, that mentions if 4K sectors are supported, but I am not able to read it as I don't have a subscription: https://access.redhat.com/solutions/56494 Its hard to believe that 4K drives have not been used from others on oVirt deployments. Alex On Tue, Jun 6, 2017 at 3:18 PM, Abi Askushi wrote: > Hi Krutika, > > My comments inline. > > Also attached the strace of: > strace -y -ff -o /root/512-trace-on-root.log dd if=/dev/zero > of=/mnt/test2.img oflag=direct bs=512 count=1 > > and of: > strace -y -ff -o /root/4096-trace-on-root.log dd if=/dev/zero > of=/mnt/test2.img oflag=direct bs=4096 count=16 > > I have mounted gluster volume at /mnt. > The dd with bs=4096 is successful. > > The gluster mount log gives only the following: > [2017-06-06 12:04:54.102576] W [MSGID: 114031] > [client-rpc-fops.c:854:client3_3_writev_cbk] > 0-engine-client-0: remote operation failed [Invalid argument] > [2017-06-06 12:04:54.102591] W [MSGID: 114031] > [client-rpc-fops.c:854:client3_3_writev_cbk] > 0-engine-client-1: remote operation failed [Invalid argument] > [2017-06-06 12:04:54.103355] W [fuse-bridge.c:2312:fuse_writev_cbk] > 0-glusterfs-fuse: 205: WRITE => -1 gfid=075ab3a5-0274-4f07-a075-2748c3b4d394 > fd=0x7faf1d08706c (Transport endpoint is not connected) > > The gluster brick log gives: > [2017-06-06 12:07:03.793080] E [MSGID: 113072] [posix.c:3453:posix_writev] > 0-engine-posix: write failed: offset 0, [Invalid argument] > [2017-06-06 12:07:03.793172] E [MSGID: 115067] > [server-rpc-fops.c:1346:server_writev_cbk] > 0-engine-server: 291: WRITEV 0 (075ab3a5-0274-4f07-a075-2748c3b4d394) ==> > (Invalid argument) [Invalid argument] > > > > On Tue, Jun 6, 2017 at 12:50 PM, Krutika Dhananjay > wrote: > >> OK. >> >> So for the 'Transport endpoint is not connected' issue, could you share >> the mount and brick logs? >> >> Hmmm.. 'Invalid argument' error even on the root partition. What if you >> change bs to 4096 and run? >> > If I use bs=4096 the dd is successful on /root and at gluster mounted > volume. > >> >> The logs I showed in my earlier mail shows that gluster is merely >> returning the error it got from the disk file system where the >> brick is hosted. But you're right about the fact that the offset 127488 >> is not 4K-aligned. >> >> If the dd on /root worked for you with bs=4096, could you try the same >> directly on gluster mount point on a dummy file and capture the strace >> output of dd? >> You can perhaps reuse your existing gluster volume by mounting it at >> another location and doing the dd. >> Here's what you need to execute: >> >> strace -ff -T -p -o >> ` >> >> FWIW, here's something I found in man(2) open: >> >> >> >> >> *Under Linux 2.4, transfer sizes, and the alignment of the user >> buffer and the file offset must all be multiples of the logical block size >> of the filesystem. Since Linux 2.6.0, alignment to the logical block size >> of the underlying storage (typically 512 bytes) suffices. The >> logical block size can be determined using the ioctl(2) BLKSSZGET operation >> or from the shell using the command: blockdev --getss* >> > Please note also that the physical disks are of 4K sector size (native). > Thus OS is having 4096/4096 local/physical sector size. > [root@v0 ~]# blockdev --getss /dev/sda > 4096 > [root@v0 ~]# blockdev --getpbsz /dev/sda > 4096 > >> >> >> -Krutika >> >> >> On Tue, Jun 6, 2017 at 1:18 AM, Abi Askushi >> wrote: >> >>> Also when testing with dd i get the following: >>> >>> *Testing on the gluster mount: * >>> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/10.100.100.1: >>> _engine/test2.img oflag=direct bs=512 count=1 >>> dd: error writing β/rhev/data-center/mnt/glusterSD/10.100.100.1: >>> _engine/test2.imgβ: *Transport endpoint is not connected* >>> 1+0 records in >>> 0+0 records out >>> 0 bytes (0 B) copied, 0.00336755 s, 0.0 kB/s >>> >>> *Testing on the /root directory (XFS): * >>> dd if=/dev/zero of=/test2.img oflag=direct bs=512 count=1 >>> dd: error writing β/test2.imgβ:* Invalid argument* >>> 1+0 records in >>> 0+0 records out >>> 0 bytes (0 B) copied, 0.000321239 s, 0.0 kB/s >>> >>> Seems that the gluster is trying to do the same and fails. >>> >>> >>> >>> On Mon, Jun 5, 2017 at 10:10 PM, Abi Askushi >>> wrote: >>> &g
Re: [ovirt-users] oVirt gluster sanlock issue
Hi Krutika, My comments inline. Also attached the strace of: strace -y -ff -o /root/512-trace-on-root.log dd if=/dev/zero of=/mnt/test2.img oflag=direct bs=512 count=1 and of: strace -y -ff -o /root/4096-trace-on-root.log dd if=/dev/zero of=/mnt/test2.img oflag=direct bs=4096 count=16 I have mounted gluster volume at /mnt. The dd with bs=4096 is successful. The gluster mount log gives only the following: [2017-06-06 12:04:54.102576] W [MSGID: 114031] [client-rpc-fops.c:854:client3_3_writev_cbk] 0-engine-client-0: remote operation failed [Invalid argument] [2017-06-06 12:04:54.102591] W [MSGID: 114031] [client-rpc-fops.c:854:client3_3_writev_cbk] 0-engine-client-1: remote operation failed [Invalid argument] [2017-06-06 12:04:54.103355] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 205: WRITE => -1 gfid=075ab3a5-0274-4f07-a075-2748c3b4d394 fd=0x7faf1d08706c (Transport endpoint is not connected) The gluster brick log gives: [2017-06-06 12:07:03.793080] E [MSGID: 113072] [posix.c:3453:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2017-06-06 12:07:03.793172] E [MSGID: 115067] [server-rpc-fops.c:1346:server_writev_cbk] 0-engine-server: 291: WRITEV 0 (075ab3a5-0274-4f07-a075-2748c3b4d394) ==> (Invalid argument) [Invalid argument] On Tue, Jun 6, 2017 at 12:50 PM, Krutika Dhananjay wrote: > OK. > > So for the 'Transport endpoint is not connected' issue, could you share > the mount and brick logs? > > Hmmm.. 'Invalid argument' error even on the root partition. What if you > change bs to 4096 and run? > If I use bs=4096 the dd is successful on /root and at gluster mounted volume. > > The logs I showed in my earlier mail shows that gluster is merely > returning the error it got from the disk file system where the > brick is hosted. But you're right about the fact that the offset 127488 is > not 4K-aligned. > > If the dd on /root worked for you with bs=4096, could you try the same > directly on gluster mount point on a dummy file and capture the strace > output of dd? > You can perhaps reuse your existing gluster volume by mounting it at > another location and doing the dd. > Here's what you need to execute: > > strace -ff -T -p -o > ` > > FWIW, here's something I found in man(2) open: > > > > > *Under Linux 2.4, transfer sizes, and the alignment of the user > buffer and the file offset must all be multiples of the logical block size > of the filesystem. Since Linux 2.6.0, alignment to the logical block size > of the underlying storage (typically 512 bytes) suffices. The > logical block size can be determined using the ioctl(2) BLKSSZGET operation > or from the shell using the command: blockdev --getss* > Please note also that the physical disks are of 4K sector size (native). Thus OS is having 4096/4096 local/physical sector size. [root@v0 ~]# blockdev --getss /dev/sda 4096 [root@v0 ~]# blockdev --getpbsz /dev/sda 4096 > > > -Krutika > > > On Tue, Jun 6, 2017 at 1:18 AM, Abi Askushi > wrote: > >> Also when testing with dd i get the following: >> >> *Testing on the gluster mount: * >> dd if=/dev/zero >> of=/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/test2.img >> oflag=direct bs=512 count=1 >> dd: error writing β/rhev/data-center/mnt/glusterSD/10.100.100.1: >> _engine/test2.imgβ: *Transport endpoint is not connected* >> 1+0 records in >> 0+0 records out >> 0 bytes (0 B) copied, 0.00336755 s, 0.0 kB/s >> >> *Testing on the /root directory (XFS): * >> dd if=/dev/zero of=/test2.img oflag=direct bs=512 count=1 >> dd: error writing β/test2.imgβ:* Invalid argument* >> 1+0 records in >> 0+0 records out >> 0 bytes (0 B) copied, 0.000321239 s, 0.0 kB/s >> >> Seems that the gluster is trying to do the same and fails. >> >> >> >> On Mon, Jun 5, 2017 at 10:10 PM, Abi Askushi >> wrote: >> >>> The question that rises is what is needed to make gluster aware of the >>> 4K physical sectors presented to it (the logical sector is also 4K). The >>> offset (127488) at the log does not seem aligned at 4K. >>> >>> Alex >>> >>> On Mon, Jun 5, 2017 at 2:47 PM, Abi Askushi >>> wrote: >>> >>>> Hi Krutika, >>>> >>>> I am saying that I am facing this issue with 4k drives. I never >>>> encountered this issue with 512 drives. >>>> >>>> Alex >>>> >>>> On Jun 5, 2017 14:26, "Krutika Dhananjay" wrote: >>>> >>>>> This seems like a case of O_DIRECT reads and writes gone wrong, >>>>> judging by the 'Invalid
Re: [ovirt-users] oVirt gluster sanlock issue
Also when testing with dd i get the following: *Testing on the gluster mount: * dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/test2.img oflag=direct bs=512 count=1 dd: error writing β/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/test2.imgβ: *Transport endpoint is not connected* 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00336755 s, 0.0 kB/s *Testing on the /root directory (XFS): * dd if=/dev/zero of=/test2.img oflag=direct bs=512 count=1 dd: error writing β/test2.imgβ:* Invalid argument* 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000321239 s, 0.0 kB/s Seems that the gluster is trying to do the same and fails. On Mon, Jun 5, 2017 at 10:10 PM, Abi Askushi wrote: > The question that rises is what is needed to make gluster aware of the 4K > physical sectors presented to it (the logical sector is also 4K). The > offset (127488) at the log does not seem aligned at 4K. > > Alex > > On Mon, Jun 5, 2017 at 2:47 PM, Abi Askushi > wrote: > >> Hi Krutika, >> >> I am saying that I am facing this issue with 4k drives. I never >> encountered this issue with 512 drives. >> >> Alex >> >> On Jun 5, 2017 14:26, "Krutika Dhananjay" wrote: >> >>> This seems like a case of O_DIRECT reads and writes gone wrong, judging >>> by the 'Invalid argument' errors. >>> >>> The two operations that have failed on gluster bricks are: >>> >>> [2017-06-05 09:40:39.428979] E [MSGID: 113072] >>> [posix.c:3453:posix_writev] 0-engine-posix: write failed: offset 0, >>> [Invalid argument] >>> [2017-06-05 09:41:00.865760] E [MSGID: 113040] >>> [posix.c:3178:posix_readv] 0-engine-posix: read failed on >>> gfid=8c94f658-ac3c-4e3a-b368-8c038513a914, fd=0x7f408584c06c, >>> offset=127488 size=512, buf=0x7f4083c0b000 [Invalid argument] >>> >>> But then, both the write and the read have 512byte-aligned offset, size >>> and buf address (which is correct). >>> >>> Are you saying you don't see this issue with 4K block-size? >>> >>> -Krutika >>> >>> On Mon, Jun 5, 2017 at 3:21 PM, Abi Askushi >>> wrote: >>> >>>> Hi Sahina, >>>> >>>> Attached are the logs. Let me know if sth else is needed. >>>> >>>> I have 5 disks (with 4K physical sector) in RAID5. The RAID has 64K >>>> stripe size at the moment. >>>> I have prepared the storage as below: >>>> >>>> pvcreate --dataalignment 256K /dev/sda4 >>>> vgcreate --physicalextentsize 256K gluster /dev/sda4 >>>> >>>> lvcreate -n engine --size 120G gluster >>>> mkfs.xfs -f -i size=512 /dev/gluster/engine >>>> >>>> Thanx, >>>> Alex >>>> >>>> On Mon, Jun 5, 2017 at 12:14 PM, Sahina Bose wrote: >>>> >>>>> Can we have the gluster mount logs and brick logs to check if it's the >>>>> same issue? >>>>> >>>>> On Sun, Jun 4, 2017 at 11:21 PM, Abi Askushi >>>>> wrote: >>>>> >>>>>> I clean installed everything and ran into the same. >>>>>> I then ran gdeploy and encountered the same issue when deploying >>>>>> engine. >>>>>> Seems that gluster (?) doesn't like 4K sector drives. I am not sure >>>>>> if it has to do with alignment. The weird thing is that gluster volumes >>>>>> are >>>>>> all ok, replicating normally and no split brain is reported. >>>>>> >>>>>> The solution to the mentioned bug (1386443 >>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format >>>>>> with 512 sector size, which for my case is not an option: >>>>>> >>>>>> mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine >>>>>> illegal sector size 512; hw sector is 4096 >>>>>> >>>>>> Is there any workaround to address this? >>>>>> >>>>>> Thanx, >>>>>> Alex >>>>>> >>>>>> >>>>>> On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi >>>>>> wrote: >>>>>> >>>>>>> Hi Maor, >>>>>>> >>>>>>> My disk are of 4K block size and from this bug seems that gluster >>>>>>> replica needs 512B block size. >>>>>>> Is there
Re: [ovirt-users] oVirt gluster sanlock issue
The question that rises is what is needed to make gluster aware of the 4K physical sectors presented to it (the logical sector is also 4K). The offset (127488) at the log does not seem aligned at 4K. Alex On Mon, Jun 5, 2017 at 2:47 PM, Abi Askushi wrote: > Hi Krutika, > > I am saying that I am facing this issue with 4k drives. I never > encountered this issue with 512 drives. > > Alex > > On Jun 5, 2017 14:26, "Krutika Dhananjay" wrote: > >> This seems like a case of O_DIRECT reads and writes gone wrong, judging >> by the 'Invalid argument' errors. >> >> The two operations that have failed on gluster bricks are: >> >> [2017-06-05 09:40:39.428979] E [MSGID: 113072] >> [posix.c:3453:posix_writev] 0-engine-posix: write failed: offset 0, >> [Invalid argument] >> [2017-06-05 09:41:00.865760] E [MSGID: 113040] [posix.c:3178:posix_readv] >> 0-engine-posix: read failed on gfid=8c94f658-ac3c-4e3a-b368-8c038513a914, >> fd=0x7f408584c06c, offset=127488 size=512, buf=0x7f4083c0b000 [Invalid >> argument] >> >> But then, both the write and the read have 512byte-aligned offset, size >> and buf address (which is correct). >> >> Are you saying you don't see this issue with 4K block-size? >> >> -Krutika >> >> On Mon, Jun 5, 2017 at 3:21 PM, Abi Askushi >> wrote: >> >>> Hi Sahina, >>> >>> Attached are the logs. Let me know if sth else is needed. >>> >>> I have 5 disks (with 4K physical sector) in RAID5. The RAID has 64K >>> stripe size at the moment. >>> I have prepared the storage as below: >>> >>> pvcreate --dataalignment 256K /dev/sda4 >>> vgcreate --physicalextentsize 256K gluster /dev/sda4 >>> >>> lvcreate -n engine --size 120G gluster >>> mkfs.xfs -f -i size=512 /dev/gluster/engine >>> >>> Thanx, >>> Alex >>> >>> On Mon, Jun 5, 2017 at 12:14 PM, Sahina Bose wrote: >>> >>>> Can we have the gluster mount logs and brick logs to check if it's the >>>> same issue? >>>> >>>> On Sun, Jun 4, 2017 at 11:21 PM, Abi Askushi >>>> wrote: >>>> >>>>> I clean installed everything and ran into the same. >>>>> I then ran gdeploy and encountered the same issue when deploying >>>>> engine. >>>>> Seems that gluster (?) doesn't like 4K sector drives. I am not sure if >>>>> it has to do with alignment. The weird thing is that gluster volumes are >>>>> all ok, replicating normally and no split brain is reported. >>>>> >>>>> The solution to the mentioned bug (1386443 >>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format >>>>> with 512 sector size, which for my case is not an option: >>>>> >>>>> mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine >>>>> illegal sector size 512; hw sector is 4096 >>>>> >>>>> Is there any workaround to address this? >>>>> >>>>> Thanx, >>>>> Alex >>>>> >>>>> >>>>> On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi >>>>> wrote: >>>>> >>>>>> Hi Maor, >>>>>> >>>>>> My disk are of 4K block size and from this bug seems that gluster >>>>>> replica needs 512B block size. >>>>>> Is there a way to make gluster function with 4K drives? >>>>>> >>>>>> Thank you! >>>>>> >>>>>> On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk >>>>>> wrote: >>>>>> >>>>>>> Hi Alex, >>>>>>> >>>>>>> I saw a bug that might be related to the issue you encountered at >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1386443 >>>>>>> >>>>>>> Sahina, maybe you have any advise? Do you think that BZ1386443is >>>>>>> related? >>>>>>> >>>>>>> Regards, >>>>>>> Maor >>>>>>> >>>>>>> On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi >>>>>>> wrote: >>>>>>> > Hi All, >>>>>>> > >>>>>>> > I have installed successfully several times oVirt (version 4.1) >>>>>>> with 3 nodes >>>>>>> > on
Re: [ovirt-users] oVirt gluster sanlock issue
Hi Krutika, I am saying that I am facing this issue with 4k drives. I never encountered this issue with 512 drives. Alex On Jun 5, 2017 14:26, "Krutika Dhananjay" wrote: > This seems like a case of O_DIRECT reads and writes gone wrong, judging by > the 'Invalid argument' errors. > > The two operations that have failed on gluster bricks are: > > [2017-06-05 09:40:39.428979] E [MSGID: 113072] [posix.c:3453:posix_writev] > 0-engine-posix: write failed: offset 0, [Invalid argument] > [2017-06-05 09:41:00.865760] E [MSGID: 113040] [posix.c:3178:posix_readv] > 0-engine-posix: read failed on gfid=8c94f658-ac3c-4e3a-b368-8c038513a914, > fd=0x7f408584c06c, offset=127488 size=512, buf=0x7f4083c0b000 [Invalid > argument] > > But then, both the write and the read have 512byte-aligned offset, size > and buf address (which is correct). > > Are you saying you don't see this issue with 4K block-size? > > -Krutika > > On Mon, Jun 5, 2017 at 3:21 PM, Abi Askushi > wrote: > >> Hi Sahina, >> >> Attached are the logs. Let me know if sth else is needed. >> >> I have 5 disks (with 4K physical sector) in RAID5. The RAID has 64K >> stripe size at the moment. >> I have prepared the storage as below: >> >> pvcreate --dataalignment 256K /dev/sda4 >> vgcreate --physicalextentsize 256K gluster /dev/sda4 >> >> lvcreate -n engine --size 120G gluster >> mkfs.xfs -f -i size=512 /dev/gluster/engine >> >> Thanx, >> Alex >> >> On Mon, Jun 5, 2017 at 12:14 PM, Sahina Bose wrote: >> >>> Can we have the gluster mount logs and brick logs to check if it's the >>> same issue? >>> >>> On Sun, Jun 4, 2017 at 11:21 PM, Abi Askushi >>> wrote: >>> >>>> I clean installed everything and ran into the same. >>>> I then ran gdeploy and encountered the same issue when deploying >>>> engine. >>>> Seems that gluster (?) doesn't like 4K sector drives. I am not sure if >>>> it has to do with alignment. The weird thing is that gluster volumes are >>>> all ok, replicating normally and no split brain is reported. >>>> >>>> The solution to the mentioned bug (1386443 >>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format >>>> with 512 sector size, which for my case is not an option: >>>> >>>> mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine >>>> illegal sector size 512; hw sector is 4096 >>>> >>>> Is there any workaround to address this? >>>> >>>> Thanx, >>>> Alex >>>> >>>> >>>> On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi >>>> wrote: >>>> >>>>> Hi Maor, >>>>> >>>>> My disk are of 4K block size and from this bug seems that gluster >>>>> replica needs 512B block size. >>>>> Is there a way to make gluster function with 4K drives? >>>>> >>>>> Thank you! >>>>> >>>>> On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk >>>>> wrote: >>>>> >>>>>> Hi Alex, >>>>>> >>>>>> I saw a bug that might be related to the issue you encountered at >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1386443 >>>>>> >>>>>> Sahina, maybe you have any advise? Do you think that BZ1386443is >>>>>> related? >>>>>> >>>>>> Regards, >>>>>> Maor >>>>>> >>>>>> On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi >>>>>> wrote: >>>>>> > Hi All, >>>>>> > >>>>>> > I have installed successfully several times oVirt (version 4.1) >>>>>> with 3 nodes >>>>>> > on top glusterfs. >>>>>> > >>>>>> > This time, when trying to configure the same setup, I am facing the >>>>>> > following issue which doesn't seem to go away. During installation >>>>>> i get the >>>>>> > error: >>>>>> > >>>>>> > Failed to execute stage 'Misc configuration': Cannot acquire host >>>>>> id: >>>>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>>>> 'Sanlock >>>>>> > lockspace add failure', 'Invalid argu
Re: [ovirt-users] oVirt gluster sanlock issue
Hi Sahina, Attached are the logs. Let me know if sth else is needed. I have 5 disks (with 4K physical sector) in RAID5. The RAID has 64K stripe size at the moment. I have prepared the storage as below: pvcreate --dataalignment 256K /dev/sda4 vgcreate --physicalextentsize 256K gluster /dev/sda4 lvcreate -n engine --size 120G gluster mkfs.xfs -f -i size=512 /dev/gluster/engine Thanx, Alex On Mon, Jun 5, 2017 at 12:14 PM, Sahina Bose wrote: > Can we have the gluster mount logs and brick logs to check if it's the > same issue? > > On Sun, Jun 4, 2017 at 11:21 PM, Abi Askushi > wrote: > >> I clean installed everything and ran into the same. >> I then ran gdeploy and encountered the same issue when deploying engine. >> Seems that gluster (?) doesn't like 4K sector drives. I am not sure if it >> has to do with alignment. The weird thing is that gluster volumes are all >> ok, replicating normally and no split brain is reported. >> >> The solution to the mentioned bug (1386443 >> <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format >> with 512 sector size, which for my case is not an option: >> >> mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine >> illegal sector size 512; hw sector is 4096 >> >> Is there any workaround to address this? >> >> Thanx, >> Alex >> >> >> On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi >> wrote: >> >>> Hi Maor, >>> >>> My disk are of 4K block size and from this bug seems that gluster >>> replica needs 512B block size. >>> Is there a way to make gluster function with 4K drives? >>> >>> Thank you! >>> >>> On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk >>> wrote: >>> >>>> Hi Alex, >>>> >>>> I saw a bug that might be related to the issue you encountered at >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1386443 >>>> >>>> Sahina, maybe you have any advise? Do you think that BZ1386443is >>>> related? >>>> >>>> Regards, >>>> Maor >>>> >>>> On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi >>>> wrote: >>>> > Hi All, >>>> > >>>> > I have installed successfully several times oVirt (version 4.1) with >>>> 3 nodes >>>> > on top glusterfs. >>>> > >>>> > This time, when trying to configure the same setup, I am facing the >>>> > following issue which doesn't seem to go away. During installation i >>>> get the >>>> > error: >>>> > >>>> > Failed to execute stage 'Misc configuration': Cannot acquire host id: >>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>> 'Sanlock >>>> > lockspace add failure', 'Invalid argument')) >>>> > >>>> > The only different in this setup is that instead of standard >>>> partitioning i >>>> > have GPT partitioning and the disks have 4K block size instead of 512. >>>> > >>>> > The /var/log/sanlock.log has the following lines: >>>> > >>>> > 2017-06-03 19:21:15+0200 23450 [943]: s9 lockspace >>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:250:/rhev/data-center/m >>>> nt/_var_lib_ovirt-hosted-engin-setup_tmptjkIDI/ba6bd862-c2b8 >>>> -46e7-b2c8-91e4a5bb2047/dom_md/ids:0 >>>> > 2017-06-03 19:21:36+0200 23471 [944]: s9:r5 resource >>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:SDM:/rhev/data-center/m >>>> nt/_var_lib_ovirt-hosted-engine-setup_tmptjkIDI/ba6bd862-c2b >>>> 8-46e7-b2c8-91e4a5bb2047/dom_md/leases:1048576 >>>> > for 2,9,23040 >>>> > 2017-06-03 19:21:36+0200 23471 [943]: s10 lockspace >>>> > a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922:250:/rhev/data-center/m >>>> nt/glusterSD/10.100.100.1:_engine/a5a6b0e7-fc3f-4838-8e26-c8 >>>> b4d5e5e922/dom_md/ids:0 >>>> > 2017-06-03 19:21:36+0200 23471 [23522]: a5a6b0e7 aio collect RD >>>> > 0x7f59b8c0:0x7f59b8d0:0x7f59b0101000 result -22:0 match res >>>> > 2017-06-03 19:21:36+0200 23471 [23522]: read_sectors delta_leader >>>> offset >>>> > 127488 rv -22 >>>> > /rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/a5a6b0e >>>> 7-fc3f-4838-8e26-c8b4d5e5e922/dom_md/ids >>>> > 2017-06-03 19:21:37+0200 23472 [930]: s9 host 250 1 23450 >>
Re: [ovirt-users] oVirt gluster sanlock issue
I clean installed everything and ran into the same. I then ran gdeploy and encountered the same issue when deploying engine. Seems that gluster (?) doesn't like 4K sector drives. I am not sure if it has to do with alignment. The weird thing is that gluster volumes are all ok, replicating normally and no split brain is reported. The solution to the mentioned bug (1386443 <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format with 512 sector size, which for my case is not an option: mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine illegal sector size 512; hw sector is 4096 Is there any workaround to address this? Thanx, Alex On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi wrote: > Hi Maor, > > My disk are of 4K block size and from this bug seems that gluster replica > needs 512B block size. > Is there a way to make gluster function with 4K drives? > > Thank you! > > On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk wrote: > >> Hi Alex, >> >> I saw a bug that might be related to the issue you encountered at >> https://bugzilla.redhat.com/show_bug.cgi?id=1386443 >> >> Sahina, maybe you have any advise? Do you think that BZ1386443is related? >> >> Regards, >> Maor >> >> On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi >> wrote: >> > Hi All, >> > >> > I have installed successfully several times oVirt (version 4.1) with 3 >> nodes >> > on top glusterfs. >> > >> > This time, when trying to configure the same setup, I am facing the >> > following issue which doesn't seem to go away. During installation i >> get the >> > error: >> > >> > Failed to execute stage 'Misc configuration': Cannot acquire host id: >> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, 'Sanlock >> > lockspace add failure', 'Invalid argument')) >> > >> > The only different in this setup is that instead of standard >> partitioning i >> > have GPT partitioning and the disks have 4K block size instead of 512. >> > >> > The /var/log/sanlock.log has the following lines: >> > >> > 2017-06-03 19:21:15+0200 23450 [943]: s9 lockspace >> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:250:/rhev/data-center/ >> mnt/_var_lib_ovirt-hosted-engin-setup_tmptjkIDI/ba6bd862 >> -c2b8-46e7-b2c8-91e4a5bb2047/dom_md/ids:0 >> > 2017-06-03 19:21:36+0200 23471 [944]: s9:r5 resource >> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:SDM:/rhev/data-center/ >> mnt/_var_lib_ovirt-hosted-engine-setup_tmptjkIDI/ba6bd86 >> 2-c2b8-46e7-b2c8-91e4a5bb2047/dom_md/leases:1048576 >> > for 2,9,23040 >> > 2017-06-03 19:21:36+0200 23471 [943]: s10 lockspace >> > a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922:250:/rhev/data-center/ >> mnt/glusterSD/10.100.100.1:_engine/a5a6b0e7-fc3f-4838- >> 8e26-c8b4d5e5e922/dom_md/ids:0 >> > 2017-06-03 19:21:36+0200 23471 [23522]: a5a6b0e7 aio collect RD >> > 0x7f59b8c0:0x7f59b8d0:0x7f59b0101000 result -22:0 match res >> > 2017-06-03 19:21:36+0200 23471 [23522]: read_sectors delta_leader offset >> > 127488 rv -22 >> > /rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/ >> a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922/dom_md/ids >> > 2017-06-03 19:21:37+0200 23472 [930]: s9 host 250 1 23450 >> > 88c2244c-a782-40ed-9560-6cfa4d46f853.v0.neptune >> > 2017-06-03 19:21:37+0200 23472 [943]: s10 add_lockspace fail result -22 >> > >> > And /var/log/vdsm/vdsm.log says: >> > >> > 2017-06-03 19:19:38,176+0200 WARN (jsonrpc/3) >> > [storage.StorageServer.MountConnection] Using user specified >> > backup-volfile-servers option (storageServer:253) >> > 2017-06-03 19:21:12,379+0200 WARN (periodic/1) [throttled] MOM not >> > available. (throttledlog:105) >> > 2017-06-03 19:21:12,380+0200 WARN (periodic/1) [throttled] MOM not >> > available, KSM stats will be missing. (throttledlog:105) >> > 2017-06-03 19:21:14,714+0200 WARN (jsonrpc/1) >> > [storage.StorageServer.MountConnection] Using user specified >> > backup-volfile-servers option (storageServer:253) >> > 2017-06-03 19:21:15,515+0200 ERROR (jsonrpc/4) [storage.initSANLock] >> Cannot >> > initialize SANLock for domain a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922 >> > (clusterlock:238) >> > Traceback (most recent call last): >> > File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", >> line >> > 234, in initSANLock >> > sanlock.init_lockspace(sdUUID, idsPath) >> >
Re: [ovirt-users] oVirt gluster sanlock issue
Hi Maor, My disk are of 4K block size and from this bug seems that gluster replica needs 512B block size. Is there a way to make gluster function with 4K drives? Thank you! On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk wrote: > Hi Alex, > > I saw a bug that might be related to the issue you encountered at > https://bugzilla.redhat.com/show_bug.cgi?id=1386443 > > Sahina, maybe you have any advise? Do you think that BZ1386443is related? > > Regards, > Maor > > On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi > wrote: > > Hi All, > > > > I have installed successfully several times oVirt (version 4.1) with 3 > nodes > > on top glusterfs. > > > > This time, when trying to configure the same setup, I am facing the > > following issue which doesn't seem to go away. During installation i get > the > > error: > > > > Failed to execute stage 'Misc configuration': Cannot acquire host id: > > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, 'Sanlock > > lockspace add failure', 'Invalid argument')) > > > > The only different in this setup is that instead of standard > partitioning i > > have GPT partitioning and the disks have 4K block size instead of 512. > > > > The /var/log/sanlock.log has the following lines: > > > > 2017-06-03 19:21:15+0200 23450 [943]: s9 lockspace > > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:250:/rhev/data- > center/mnt/_var_lib_ovirt-hosted-engin-setup_tmptjkIDI/ > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047/dom_md/ids:0 > > 2017-06-03 19:21:36+0200 23471 [944]: s9:r5 resource > > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:SDM:/rhev/data- > center/mnt/_var_lib_ovirt-hosted-engine-setup_tmptjkIDI/ > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047/dom_md/leases:1048576 > > for 2,9,23040 > > 2017-06-03 19:21:36+0200 23471 [943]: s10 lockspace > > a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922:250:/rhev/data- > center/mnt/glusterSD/10.100.100.1:_engine/a5a6b0e7-fc3f- > 4838-8e26-c8b4d5e5e922/dom_md/ids:0 > > 2017-06-03 19:21:36+0200 23471 [23522]: a5a6b0e7 aio collect RD > > 0x7f59b8c0:0x7f59b8d0:0x7f59b0101000 result -22:0 match res > > 2017-06-03 19:21:36+0200 23471 [23522]: read_sectors delta_leader offset > > 127488 rv -22 > > /rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/a5a6b0e7-fc3f-4838- > 8e26-c8b4d5e5e922/dom_md/ids > > 2017-06-03 19:21:37+0200 23472 [930]: s9 host 250 1 23450 > > 88c2244c-a782-40ed-9560-6cfa4d46f853.v0.neptune > > 2017-06-03 19:21:37+0200 23472 [943]: s10 add_lockspace fail result -22 > > > > And /var/log/vdsm/vdsm.log says: > > > > 2017-06-03 19:19:38,176+0200 WARN (jsonrpc/3) > > [storage.StorageServer.MountConnection] Using user specified > > backup-volfile-servers option (storageServer:253) > > 2017-06-03 19:21:12,379+0200 WARN (periodic/1) [throttled] MOM not > > available. (throttledlog:105) > > 2017-06-03 19:21:12,380+0200 WARN (periodic/1) [throttled] MOM not > > available, KSM stats will be missing. (throttledlog:105) > > 2017-06-03 19:21:14,714+0200 WARN (jsonrpc/1) > > [storage.StorageServer.MountConnection] Using user specified > > backup-volfile-servers option (storageServer:253) > > 2017-06-03 19:21:15,515+0200 ERROR (jsonrpc/4) [storage.initSANLock] > Cannot > > initialize SANLock for domain a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922 > > (clusterlock:238) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", > line > > 234, in initSANLock > > sanlock.init_lockspace(sdUUID, idsPath) > > SanlockException: (107, 'Sanlock lockspace init failure', 'Transport > > endpoint is not connected') > > 2017-06-03 19:21:15,515+0200 WARN (jsonrpc/4) > > [storage.StorageDomainManifest] lease did not initialize successfully > > (sd:557) > > Traceback (most recent call last): > > File "/usr/share/vdsm/storage/sd.py", line 552, in initDomainLock > > self._domainLock.initLock(self.getDomainLease()) > > File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", > line > > 271, in initLock > > initSANLock(self._sdUUID, self._idsPath, lease) > > File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", > line > > 239, in initSANLock > > raise se.ClusterLockInitError() > > ClusterLockInitError: Could not initialize cluster lock: () > > 2017-06-03 19:21:37,867+0200 ERROR (jsonrpc/2) [storage.StoragePool] > Create > > pool hosted_datacenter canceled (sp:655) > > Traceback (most recent cal
[ovirt-users] oVirt gluster sanlock issue
Hi All, I have installed successfully several times oVirt (version 4.1) with 3 nodes on top glusterfs. This time, when trying to configure the same setup, I am facing the following issue which doesn't seem to go away. During installation i get the error: Failed to execute stage 'Misc configuration': Cannot acquire host id: (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument')) The only different in this setup is that instead of standard partitioning i have GPT partitioning and the disks have 4K block size instead of 512. The /var/log/sanlock.log has the following lines: 2017-06-03 19:21:15+0200 23450 [943]: s9 lockspace ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:250:/rhev/data-center/mnt/_var_lib_ovirt-hosted-engin-setup_tmptjkIDI/ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047/dom_md/ids:0 2017-06-03 19:21:36+0200 23471 [944]: s9:r5 resource ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:SDM:/rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmptjkIDI/ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047/dom_md/leases:1048576 for 2,9,23040 2017-06-03 19:21:36+0200 23471 [943]: s10 lockspace a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922:250:/rhev/data-center/mnt/glusterSD/10.100.100.1: _engine/a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922/dom_md/ids:0 2017-06-03 19:21:36+0200 23471 [23522]: a5a6b0e7 aio collect RD 0x7f59b8c0:0x7f59b8d0:0x7f59b0101000 result -22:0 match res 2017-06-03 19:21:36+0200 23471 [23522]: read_sectors delta_leader offset 127488 rv -22 /rhev/data-center/mnt/glusterSD/10.100.100.1: _engine/a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922/dom_md/ids 2017-06-03 19:21:37+0200 23472 [930]: s9 host 250 1 23450 88c2244c-a782-40ed-9560-6cfa4d46f853.v0.neptune 2017-06-03 19:21:37+0200 23472 [943]: s10 add_lockspace fail result -22 And /var/log/vdsm/vdsm.log says: 2017-06-03 19:19:38,176+0200 WARN (jsonrpc/3) [storage.StorageServer.MountConnection] Using user specified backup-volfile-servers option (storageServer:253) 2017-06-03 19:21:12,379+0200 WARN (periodic/1) [throttled] MOM not available. (throttledlog:105) 2017-06-03 19:21:12,380+0200 WARN (periodic/1) [throttled] MOM not available, KSM stats will be missing. (throttledlog:105) 2017-06-03 19:21:14,714+0200 WARN (jsonrpc/1) [storage.StorageServer.MountConnection] Using user specified backup-volfile-servers option (storageServer:253) 2017-06-03 19:21:15,515+0200 ERROR (jsonrpc/4) [storage.initSANLock] Cannot initialize SANLock for domain a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922 (clusterlock:238) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 234, in initSANLock sanlock.init_lockspace(sdUUID, idsPath) SanlockException: (107, 'Sanlock lockspace init failure', 'Transport endpoint is not connected') 2017-06-03 19:21:15,515+0200 WARN (jsonrpc/4) [storage.StorageDomainManifest] lease did not initialize successfully (sd:557) Traceback (most recent call last): File "/usr/share/vdsm/storage/sd.py", line 552, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 271, in initLock initSANLock(self._sdUUID, self._idsPath, lease) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 239, in initSANLock raise se.ClusterLockInitError() ClusterLockInitError: Could not initialize cluster lock: () 2017-06-03 19:21:37,867+0200 ERROR (jsonrpc/2) [storage.StoragePool] Create pool hosted_datacenter canceled (sp:655) Traceback (most recent call last): File "/usr/share/vdsm/storage/sp.py", line 652, in create self.attachSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 971, in attachSD dom.acquireHostId(self.id) File "/usr/share/vdsm/storage/sd.py", line 790, in acquireHostId self._manifest.acquireHostId(hostId, async) File "/usr/share/vdsm/storage/sd.py", line 449, in acquireHostId self._domainLock.acquireHostId(hostId, async) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 297, in acquireHostId raise se.AcquireHostIdFailure(self._sdUUID, e) AcquireHostIdFailure: Cannot acquire host id: (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument')) 2017-06-03 19:21:37,870+0200 ERROR (jsonrpc/2) [storage.StoragePool] Domain ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047 detach from MSD ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047 Ver 1 failed. (sp:528) Traceback (most recent call last): File "/usr/share/vdsm/storage/sp.py", line 525, in __cleanupDomains self.detachSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 1046, in detachSD raise se.CannotDetachMasterStorageDomain(sdUUID) CannotDetachMaster
[ovirt-users] oVirt Power Management with Cisco UCS C220 M4S
Hi All, For anyone that might stumble on a Cisco UCS C220 M4S and wondering how to configure power management. below are the steps to configure it, as it took me some hours to figure it out... 1. enable IPMI on server. (Cisco has this documented) 2. at ovirt GUI, edit host -> power management, then select "ipmilan" and add *lanplus=1* as an option. (the bold one was the tricky part) To test from command line: ipmitool -I lanplus -H -U admin -P somepass -v chassis power status It will give the response: "Chassis Power is on" Alex ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] oVirt global utilization trend
Hi, I see that the global utilization 24h trend graph (CPU, RAM, Disk) at dashboard is showing only the last 3 hours. The line for previous hours is flat (see attached). I remember that on a previous installation I had tested it was showing the full 24 hours. Tried to run again engine-setup but issue remains. Any suggestions? Thanx, Alex ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
