[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Darrell Budic]

Well, now that I’ve gone and read through that bug again in detail, I’m not 
sure I’ve worked around it after all. I do seem to recall additional discussion 
on the original bug for HA engine ligfapi a mention that RR-DNS would work to 
resolve the issue, but can’t remember the bug ID at the moment. I will test 
thoroughly the next time I update my glusterfs servers. But I firmly believe 
that I’ve never encountered that issue in over 3 years of running gluster with 
libgfapi enabled.. 

I use round robin DNS, and in theory, QEMU retries until it gets a working 
server. I also have said DNS setup in host files on all my hosts and gluster 
servers, having discovered the hard way that when your DNS server runs on an 
ovirt managed VM, you have a bootstrap problem when thing break badly :) 
Somewhere around gluster 3.12, I added backup servers to the mount options for 
my gluster storage volumes as well, and have’t had any issues with that.

And to be frank, the significant performance bonus from libgfapi is still 
absolutely worth it to me even if it means automatic HA won’t work if one 
particular server is down. I can always intervene in the DNS on my hosts if I 
have to, and it just hasn’t come up yet. 

  -Darrell


> On Feb 13, 2020, at 5:19 PM, Strahil Nikolov  wrote:
> 
> On February 13, 2020 11:51:41 PM GMT+02:00, Stephen Panicho 
> mailto:s.pani...@gmail.com>> wrote:
>> Darrell, would you care to elaborate on your HA workaround?
>> 
>> As far as I understand, only the primary Gluster host is visible to
>> libvirt
>> when using gfapi, so if that host goes down, all VMs break. I imagine
>> you're using a round-robin DNS entry for the primary Gluster host, but
>> I'd
>> like to be sure.
>> 
>> On Wed, Feb 12, 2020 at 11:01 AM Darrell Budic 
>> wrote:
>> 
>>> Yes. I’m using libgfapi access on gluster 6.7 with overt 4.3.8 just
>> fine,
>>> but I don’t use snapshots. You can work around the HA issue with DNS
>> and
>>> backup server entries on the storage domain as well. Worth it to me
>> for the
>>> performance, YMMV.
>>> 
>>> On Feb 12, 2020, at 8:04 AM, Jayme  wrote:
>>> 
>>> From my understanding it's not a default option but many users are
>> still
>>> using libgfapi successfully. I'm not sure about its status in the
>> latest
>>> 4.3.8 release but I know it is/was working for people in previous
>> versions.
>>> The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but
>> it
>>> should still be working otherwise, unless like I said something
>> changed in
>>> more recent releases of oVirt.
>>> 
>>> On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese <
>>> guillaume.pav...@interactiv-group.com> wrote:
>>> 
 Libgfapi is not supported because of an old bug in qemu. That qemu
>> bug is
 slowly getting fixed, but the bugs about Libgfapi support in ovirt
>> have
 since been closed as WONTFIX and DEFERRED
 
 See :
 https://bugzilla.redhat.com/show_bug.cgi?id=1465810
 https://bugzilla.redhat.com/show_bug.cgi?id=1484660
 > : "No plans to
 enable libgfapi in RHHI-V for now. Closing this bug"
 https://bugzilla.redhat.com/show_bug.cgi?id=1484227 
  : "No plans to
 enable libgfapi in RHHI-V for now. Closing this bug"
 https://bugzilla.redhat.com/show_bug.cgi?id=1633642 
  : "Closing this
>> as
 no action taken from long back.Please reopen if required."
 
 Would be nice if someone could reopen the closed bugs so this
>> feature
 doesn't get forgotten
 
 Guillaume Pavese
 Ingénieur Système et Réseau
 Interactiv-Group
 
 
 On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho
>> mailto:s.pani...@gmail.com>>
 wrote:
 
> I used the cockpit-based hc setup and "option
>> rpc-auth-allow-insecure"
> is absent from /etc/glusterfs/glusterd.vol.
> 
> I'm going to redo the cluster this week and report back. Thanks for
>> the
> tip!
> 
> On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic
>> mailto:bu...@onholyground.com>>
> wrote:
> 
>> The hosts will still mount the volume via FUSE, but you might
>> double
>> check you set the storage up as Gluster and not NFS.
>> 
>> Then gluster used to need some config in glusterd.vol to set
>> 
>>option rpc-auth-allow-insecure on
>> 
>> I’m not sure if that got added to a hyper converged setup or not,
>> but
>> I’d check it.
>> 
>> On Feb 10, 2020, at 4:41 PM, Stephen Panicho > >
>> wrote:
>> 
>> No, this was a relatively new cluster-- only a couple days old.
>> Just a
>> handful of VMs including the engine.
>> 
>> On Mon, Feb 10, 2020 at 5:26 PM Jayme > > wrote:
>> 
>>> Curious do the vms have 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Strahil Nikolov]

On February 13, 2020 11:51:41 PM GMT+02:00, Stephen Panicho 
 wrote:
>Darrell, would you care to elaborate on your HA workaround?
>
>As far as I understand, only the primary Gluster host is visible to
>libvirt
>when using gfapi, so if that host goes down, all VMs break. I imagine
>you're using a round-robin DNS entry for the primary Gluster host, but
>I'd
>like to be sure.
>
>On Wed, Feb 12, 2020 at 11:01 AM Darrell Budic 
>wrote:
>
>> Yes. I’m using libgfapi access on gluster 6.7 with overt 4.3.8 just
>fine,
>> but I don’t use snapshots. You can work around the HA issue with DNS
>and
>> backup server entries on the storage domain as well. Worth it to me
>for the
>> performance, YMMV.
>>
>> On Feb 12, 2020, at 8:04 AM, Jayme  wrote:
>>
>> From my understanding it's not a default option but many users are
>still
>> using libgfapi successfully. I'm not sure about its status in the
>latest
>> 4.3.8 release but I know it is/was working for people in previous
>versions.
>> The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but
>it
>> should still be working otherwise, unless like I said something
>changed in
>> more recent releases of oVirt.
>>
>> On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese <
>> guillaume.pav...@interactiv-group.com> wrote:
>>
>>> Libgfapi is not supported because of an old bug in qemu. That qemu
>bug is
>>> slowly getting fixed, but the bugs about Libgfapi support in ovirt
>have
>>> since been closed as WONTFIX and DEFERRED
>>>
>>> See :
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1465810
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1484660
>>>  : "No plans to
>>> enable libgfapi in RHHI-V for now. Closing this bug"
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1484227 : "No plans to
>>> enable libgfapi in RHHI-V for now. Closing this bug"
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1633642 : "Closing this
>as
>>> no action taken from long back.Please reopen if required."
>>>
>>> Would be nice if someone could reopen the closed bugs so this
>feature
>>> doesn't get forgotten
>>>
>>> Guillaume Pavese
>>> Ingénieur Système et Réseau
>>> Interactiv-Group
>>>
>>>
>>> On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho
>
>>> wrote:
>>>
 I used the cockpit-based hc setup and "option
>rpc-auth-allow-insecure"
 is absent from /etc/glusterfs/glusterd.vol.

 I'm going to redo the cluster this week and report back. Thanks for
>the
 tip!

 On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic
>
 wrote:

> The hosts will still mount the volume via FUSE, but you might
>double
> check you set the storage up as Gluster and not NFS.
>
> Then gluster used to need some config in glusterd.vol to set
>
> option rpc-auth-allow-insecure on
>
> I’m not sure if that got added to a hyper converged setup or not,
>but
> I’d check it.
>
> On Feb 10, 2020, at 4:41 PM, Stephen Panicho 
> wrote:
>
> No, this was a relatively new cluster-- only a couple days old.
>Just a
> handful of VMs including the engine.
>
> On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:
>
>> Curious do the vms have active snapshots?
>>
>> On Mon, Feb 10, 2020 at 5:59 PM  wrote:
>>
>>> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster
>>> running on CentOS 7.7 hosts. I was investigating poor Gluster
>performance
>>> and heard about libgfapi, so I thought I'd give it a shot.
>Looking through
>>> the documentation, followed by lots of threads and BZ reports,
>I've done
>>> the following to enable it:
>>>
>>> First, I shut down all VMs except the engine. Then...
>>>
>>> On the hosts:
>>> 1. setsebool -P virt_use_glusterfs on
>>> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>>>
>>> On the engine VM:
>>> 1. engine-config -s LibgfApiSupported=true --cver=4.3
>>> 2. systemctl restart ovirt-engine
>>>
>>> VMs now fail to launch. Am I doing this correctly? I should also
>note
>>> that the hosts still have the Gluster domain mounted via FUSE.
>>>
>>> Here's a relevant bit from engine.log:
>>>
>>> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
>>>
>node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
>>> Could not read qcow2 header: Invalid argument.
>>>
>>> The full engine.log from one of the attempts:
>>>
>>> 2020-02-06 16:38:24,909Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (ForkJoinPool-1-worker-12) [] add VM
>>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun
>treatment
>>> 2020-02-06 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Stephen Panicho]

Darrell, would you care to elaborate on your HA workaround?

As far as I understand, only the primary Gluster host is visible to libvirt
when using gfapi, so if that host goes down, all VMs break. I imagine
you're using a round-robin DNS entry for the primary Gluster host, but I'd
like to be sure.

On Wed, Feb 12, 2020 at 11:01 AM Darrell Budic 
wrote:

> Yes. I’m using libgfapi access on gluster 6.7 with overt 4.3.8 just fine,
> but I don’t use snapshots. You can work around the HA issue with DNS and
> backup server entries on the storage domain as well. Worth it to me for the
> performance, YMMV.
>
> On Feb 12, 2020, at 8:04 AM, Jayme  wrote:
>
> From my understanding it's not a default option but many users are still
> using libgfapi successfully. I'm not sure about its status in the latest
> 4.3.8 release but I know it is/was working for people in previous versions.
> The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but it
> should still be working otherwise, unless like I said something changed in
> more recent releases of oVirt.
>
> On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese <
> guillaume.pav...@interactiv-group.com> wrote:
>
>> Libgfapi is not supported because of an old bug in qemu. That qemu bug is
>> slowly getting fixed, but the bugs about Libgfapi support in ovirt have
>> since been closed as WONTFIX and DEFERRED
>>
>> See :
>> https://bugzilla.redhat.com/show_bug.cgi?id=1465810
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484660
>>  : "No plans to
>> enable libgfapi in RHHI-V for now. Closing this bug"
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484227 : "No plans to
>> enable libgfapi in RHHI-V for now. Closing this bug"
>> https://bugzilla.redhat.com/show_bug.cgi?id=1633642 : "Closing this as
>> no action taken from long back.Please reopen if required."
>>
>> Would be nice if someone could reopen the closed bugs so this feature
>> doesn't get forgotten
>>
>> Guillaume Pavese
>> Ingénieur Système et Réseau
>> Interactiv-Group
>>
>>
>> On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho 
>> wrote:
>>
>>> I used the cockpit-based hc setup and "option rpc-auth-allow-insecure"
>>> is absent from /etc/glusterfs/glusterd.vol.
>>>
>>> I'm going to redo the cluster this week and report back. Thanks for the
>>> tip!
>>>
>>> On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic 
>>> wrote:
>>>
 The hosts will still mount the volume via FUSE, but you might double
 check you set the storage up as Gluster and not NFS.

 Then gluster used to need some config in glusterd.vol to set

 option rpc-auth-allow-insecure on

 I’m not sure if that got added to a hyper converged setup or not, but
 I’d check it.

 On Feb 10, 2020, at 4:41 PM, Stephen Panicho 
 wrote:

 No, this was a relatively new cluster-- only a couple days old. Just a
 handful of VMs including the engine.

 On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:

> Curious do the vms have active snapshots?
>
> On Mon, Feb 10, 2020 at 5:59 PM  wrote:
>
>> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster
>> running on CentOS 7.7 hosts. I was investigating poor Gluster performance
>> and heard about libgfapi, so I thought I'd give it a shot. Looking 
>> through
>> the documentation, followed by lots of threads and BZ reports, I've done
>> the following to enable it:
>>
>> First, I shut down all VMs except the engine. Then...
>>
>> On the hosts:
>> 1. setsebool -P virt_use_glusterfs on
>> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>>
>> On the engine VM:
>> 1. engine-config -s LibgfApiSupported=true --cver=4.3
>> 2. systemctl restart ovirt-engine
>>
>> VMs now fail to launch. Am I doing this correctly? I should also note
>> that the hosts still have the Gluster domain mounted via FUSE.
>>
>> Here's a relevant bit from engine.log:
>>
>> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
>> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
>> Could not read qcow2 header: Invalid argument.
>>
>> The full engine.log from one of the attempts:
>>
>> 2020-02-06 16:38:24,909Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (ForkJoinPool-1-worker-12) [] add VM
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
>> 2020-02-06 16:38:25,010Z ERROR
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
>> (ForkJoinPool-1-worker-12) [] Rerun VM
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
>> 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Jayme]

I really wish these bugs would get more attention. I struggle to understand
why this isn't a priority given the performance increase's people are
reporting when switching to libgfapi. No snapshots is a deal breaker for me
unfortunately.

On Wed, Feb 12, 2020 at 12:01 PM Darrell Budic 
wrote:

> Yes. I’m using libgfapi access on gluster 6.7 with overt 4.3.8 just fine,
> but I don’t use snapshots. You can work around the HA issue with DNS and
> backup server entries on the storage domain as well. Worth it to me for the
> performance, YMMV.
>
> On Feb 12, 2020, at 8:04 AM, Jayme  wrote:
>
> From my understanding it's not a default option but many users are still
> using libgfapi successfully. I'm not sure about its status in the latest
> 4.3.8 release but I know it is/was working for people in previous versions.
> The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but it
> should still be working otherwise, unless like I said something changed in
> more recent releases of oVirt.
>
> On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese <
> guillaume.pav...@interactiv-group.com> wrote:
>
>> Libgfapi is not supported because of an old bug in qemu. That qemu bug is
>> slowly getting fixed, but the bugs about Libgfapi support in ovirt have
>> since been closed as WONTFIX and DEFERRED
>>
>> See :
>> https://bugzilla.redhat.com/show_bug.cgi?id=1465810
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484660
>>  : "No plans to
>> enable libgfapi in RHHI-V for now. Closing this bug"
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484227 : "No plans to
>> enable libgfapi in RHHI-V for now. Closing this bug"
>> https://bugzilla.redhat.com/show_bug.cgi?id=1633642 : "Closing this as
>> no action taken from long back.Please reopen if required."
>>
>> Would be nice if someone could reopen the closed bugs so this feature
>> doesn't get forgotten
>>
>> Guillaume Pavese
>> Ingénieur Système et Réseau
>> Interactiv-Group
>>
>>
>> On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho 
>> wrote:
>>
>>> I used the cockpit-based hc setup and "option rpc-auth-allow-insecure"
>>> is absent from /etc/glusterfs/glusterd.vol.
>>>
>>> I'm going to redo the cluster this week and report back. Thanks for the
>>> tip!
>>>
>>> On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic 
>>> wrote:
>>>
 The hosts will still mount the volume via FUSE, but you might double
 check you set the storage up as Gluster and not NFS.

 Then gluster used to need some config in glusterd.vol to set

 option rpc-auth-allow-insecure on

 I’m not sure if that got added to a hyper converged setup or not, but
 I’d check it.

 On Feb 10, 2020, at 4:41 PM, Stephen Panicho 
 wrote:

 No, this was a relatively new cluster-- only a couple days old. Just a
 handful of VMs including the engine.

 On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:

> Curious do the vms have active snapshots?
>
> On Mon, Feb 10, 2020 at 5:59 PM  wrote:
>
>> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster
>> running on CentOS 7.7 hosts. I was investigating poor Gluster performance
>> and heard about libgfapi, so I thought I'd give it a shot. Looking 
>> through
>> the documentation, followed by lots of threads and BZ reports, I've done
>> the following to enable it:
>>
>> First, I shut down all VMs except the engine. Then...
>>
>> On the hosts:
>> 1. setsebool -P virt_use_glusterfs on
>> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>>
>> On the engine VM:
>> 1. engine-config -s LibgfApiSupported=true --cver=4.3
>> 2. systemctl restart ovirt-engine
>>
>> VMs now fail to launch. Am I doing this correctly? I should also note
>> that the hosts still have the Gluster domain mounted via FUSE.
>>
>> Here's a relevant bit from engine.log:
>>
>> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
>> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
>> Could not read qcow2 header: Invalid argument.
>>
>> The full engine.log from one of the attempts:
>>
>> 2020-02-06 16:38:24,909Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (ForkJoinPool-1-worker-12) [] add VM
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
>> 2020-02-06 16:38:25,010Z ERROR
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
>> (ForkJoinPool-1-worker-12) [] Rerun VM
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
>> node2.ovirt.trashnet.xyz'
>> 2020-02-06 16:38:25,091Z WARN

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Darrell Budic]

Yes. I’m using libgfapi access on gluster 6.7 with overt 4.3.8 just fine, but I 
don’t use snapshots. You can work around the HA issue with DNS and backup 
server entries on the storage domain as well. Worth it to me for the 
performance, YMMV.

> On Feb 12, 2020, at 8:04 AM, Jayme  wrote:
> 
> From my understanding it's not a default option but many users are still 
> using libgfapi successfully. I'm not sure about its status in the latest 
> 4.3.8 release but I know it is/was working for people in previous versions. 
> The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but it 
> should still be working otherwise, unless like I said something changed in 
> more recent releases of oVirt.
> 
> On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese 
>  > wrote:
> Libgfapi is not supported because of an old bug in qemu. That qemu bug is 
> slowly getting fixed, but the bugs about Libgfapi support in ovirt have since 
> been closed as WONTFIX and DEFERRED
> 
> See :
> https://bugzilla.redhat.com/show_bug.cgi?id=1465810 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1484660 
>  : "No plans to enable 
> libgfapi in RHHI-V for now. Closing this bug"
> https://bugzilla.redhat.com/show_bug.cgi?id=1484227 
>  : "No plans to enable 
> libgfapi in RHHI-V for now. Closing this bug"
> https://bugzilla.redhat.com/show_bug.cgi?id=1633642 
>  : "Closing this as no 
> action taken from long back.Please reopen if required."
> 
> Would be nice if someone could reopen the closed bugs so this feature doesn't 
> get forgotten
> 
> Guillaume Pavese
> Ingénieur Système et Réseau
> Interactiv-Group
> 
> 
> On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho  > wrote:
> I used the cockpit-based hc setup and "option rpc-auth-allow-insecure" is 
> absent from /etc/glusterfs/glusterd.vol.
> 
> I'm going to redo the cluster this week and report back. Thanks for the tip!
> 
> On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic  > wrote:
> The hosts will still mount the volume via FUSE, but you might double check 
> you set the storage up as Gluster and not NFS.
> 
> Then gluster used to need some config in glusterd.vol to set 
> 
> option rpc-auth-allow-insecure on
> 
> I’m not sure if that got added to a hyper converged setup or not, but I’d 
> check it.
> 
>> On Feb 10, 2020, at 4:41 PM, Stephen Panicho > > wrote:
>> 
>> No, this was a relatively new cluster-- only a couple days old. Just a 
>> handful of VMs including the engine.
>> 
>> On Mon, Feb 10, 2020 at 5:26 PM Jayme > > wrote:
>> Curious do the vms have active snapshots?
>> 
>> On Mon, Feb 10, 2020 at 5:59 PM > > wrote:
>> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running on 
>> CentOS 7.7 hosts. I was investigating poor Gluster performance and heard 
>> about libgfapi, so I thought I'd give it a shot. Looking through the 
>> documentation, followed by lots of threads and BZ reports, I've done the 
>> following to enable it:
>> 
>> First, I shut down all VMs except the engine. Then...
>> 
>> On the hosts:
>> 1. setsebool -P virt_use_glusterfs on
>> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>> 
>> On the engine VM:
>> 1. engine-config -s LibgfApiSupported=true --cver=4.3
>> 2. systemctl restart ovirt-engine
>> 
>> VMs now fail to launch. Am I doing this correctly? I should also note that 
>> the hosts still have the Gluster domain mounted via FUSE.
>> 
>> Here's a relevant bit from engine.log:
>> 
>> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive 
>> file=gluster://node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native
>>  
>> :
>>  Could not read qcow2 header: Invalid argument.
>> 
>> The full engine.log from one of the attempts:
>> 
>> 2020-02-06 16:38:24,909Z INFO  
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] 
>> (ForkJoinPool-1-worker-12) [] add VM 
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
>> 2020-02-06 16:38:25,010Z ERROR 
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Jayme]

>From my understanding it's not a default option but many users are still
using libgfapi successfully. I'm not sure about its status in the latest
4.3.8 release but I know it is/was working for people in previous versions.
The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but it
should still be working otherwise, unless like I said something changed in
more recent releases of oVirt.

On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese <
guillaume.pav...@interactiv-group.com> wrote:

> Libgfapi is not supported because of an old bug in qemu. That qemu bug is
> slowly getting fixed, but the bugs about Libgfapi support in ovirt have
> since been closed as WONTFIX and DEFERRED
>
> See :
> https://bugzilla.redhat.com/show_bug.cgi?id=1465810
> https://bugzilla.redhat.com/show_bug.cgi?id=1484660
>  : "No plans to
> enable libgfapi in RHHI-V for now. Closing this bug"
> https://bugzilla.redhat.com/show_bug.cgi?id=1484227 : "No plans to enable
> libgfapi in RHHI-V for now. Closing this bug"
> https://bugzilla.redhat.com/show_bug.cgi?id=1633642 : "Closing this as no
> action taken from long back.Please reopen if required."
>
> Would be nice if someone could reopen the closed bugs so this feature
> doesn't get forgotten
>
> Guillaume Pavese
> Ingénieur Système et Réseau
> Interactiv-Group
>
>
> On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho 
> wrote:
>
>> I used the cockpit-based hc setup and "option rpc-auth-allow-insecure"
>> is absent from /etc/glusterfs/glusterd.vol.
>>
>> I'm going to redo the cluster this week and report back. Thanks for the
>> tip!
>>
>> On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic 
>> wrote:
>>
>>> The hosts will still mount the volume via FUSE, but you might double
>>> check you set the storage up as Gluster and not NFS.
>>>
>>> Then gluster used to need some config in glusterd.vol to set
>>>
>>> option rpc-auth-allow-insecure on
>>>
>>> I’m not sure if that got added to a hyper converged setup or not, but
>>> I’d check it.
>>>
>>> On Feb 10, 2020, at 4:41 PM, Stephen Panicho 
>>> wrote:
>>>
>>> No, this was a relatively new cluster-- only a couple days old. Just a
>>> handful of VMs including the engine.
>>>
>>> On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:
>>>
 Curious do the vms have active snapshots?

 On Mon, Feb 10, 2020 at 5:59 PM  wrote:

> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running
> on CentOS 7.7 hosts. I was investigating poor Gluster performance and 
> heard
> about libgfapi, so I thought I'd give it a shot. Looking through the
> documentation, followed by lots of threads and BZ reports, I've done the
> following to enable it:
>
> First, I shut down all VMs except the engine. Then...
>
> On the hosts:
> 1. setsebool -P virt_use_glusterfs on
> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>
> On the engine VM:
> 1. engine-config -s LibgfApiSupported=true --cver=4.3
> 2. systemctl restart ovirt-engine
>
> VMs now fail to launch. Am I doing this correctly? I should also note
> that the hosts still have the Gluster domain mounted via FUSE.
>
> Here's a relevant bit from engine.log:
>
> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
> Could not read qcow2 header: Invalid argument.
>
> The full engine.log from one of the attempts:
>
> 2020-02-06 16:38:24,909Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-12) [] add VM
> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
> 2020-02-06 16:38:25,010Z ERROR
> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
> (ForkJoinPool-1-worker-12) [] Rerun VM
> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
> node2.ovirt.trashnet.xyz'
> 2020-02-06 16:38:25,091Z WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID:
> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host
> node2.ovirt.trashnet.xyz.
> 2020-02-06 16:38:25,166Z INFO
> [org.ovirt.engine.core.bll.RunVmCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object
> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]',
> sharedLocks=''}'
> 2020-02-06 16:38:25,179Z INFO
> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
> 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Guillaume Pavese]

Libgfapi is not supported because of an old bug in qemu. That qemu bug is
slowly getting fixed, but the bugs about Libgfapi support in ovirt have
since been closed as WONTFIX and DEFERRED

See :
https://bugzilla.redhat.com/show_bug.cgi?id=1465810
https://bugzilla.redhat.com/show_bug.cgi?id=1484660
 : "No plans to enable
libgfapi in RHHI-V for now. Closing this bug"
https://bugzilla.redhat.com/show_bug.cgi?id=1484227 : "No plans to enable
libgfapi in RHHI-V for now. Closing this bug"
https://bugzilla.redhat.com/show_bug.cgi?id=1633642 : "Closing this as no
action taken from long back.Please reopen if required."

Would be nice if someone could reopen the closed bugs so this feature
doesn't get forgotten

Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho  wrote:

> I used the cockpit-based hc setup and "option rpc-auth-allow-insecure" is
> absent from /etc/glusterfs/glusterd.vol.
>
> I'm going to redo the cluster this week and report back. Thanks for the
> tip!
>
> On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic 
> wrote:
>
>> The hosts will still mount the volume via FUSE, but you might double
>> check you set the storage up as Gluster and not NFS.
>>
>> Then gluster used to need some config in glusterd.vol to set
>>
>> option rpc-auth-allow-insecure on
>>
>> I’m not sure if that got added to a hyper converged setup or not, but I’d
>> check it.
>>
>> On Feb 10, 2020, at 4:41 PM, Stephen Panicho  wrote:
>>
>> No, this was a relatively new cluster-- only a couple days old. Just a
>> handful of VMs including the engine.
>>
>> On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:
>>
>>> Curious do the vms have active snapshots?
>>>
>>> On Mon, Feb 10, 2020 at 5:59 PM  wrote:
>>>
 Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running
 on CentOS 7.7 hosts. I was investigating poor Gluster performance and heard
 about libgfapi, so I thought I'd give it a shot. Looking through the
 documentation, followed by lots of threads and BZ reports, I've done the
 following to enable it:

 First, I shut down all VMs except the engine. Then...

 On the hosts:
 1. setsebool -P virt_use_glusterfs on
 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf

 On the engine VM:
 1. engine-config -s LibgfApiSupported=true --cver=4.3
 2. systemctl restart ovirt-engine

 VMs now fail to launch. Am I doing this correctly? I should also note
 that the hosts still have the Gluster domain mounted via FUSE.

 Here's a relevant bit from engine.log:

 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
 node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
 Could not read qcow2 header: Invalid argument.

 The full engine.log from one of the attempts:

 2020-02-06 16:38:24,909Z INFO
 [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
 (ForkJoinPool-1-worker-12) [] add VM
 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
 2020-02-06 16:38:25,010Z ERROR
 [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
 (ForkJoinPool-1-worker-12) [] Rerun VM
 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
 node2.ovirt.trashnet.xyz'
 2020-02-06 16:38:25,091Z WARN
 [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
 (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID:
 USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host
 node2.ovirt.trashnet.xyz.
 2020-02-06 16:38:25,166Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
 (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object
 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]',
 sharedLocks=''}'
 2020-02-06 16:38:25,179Z INFO
 [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
 (EE-ManagedThreadFactory-engine-Thread-216) [] START,
 IsVmDuringInitiatingVDSCommand(
 IsVmDuringInitiatingVDSCommandParameters:{vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'}),
 log id: 2107f52a
 2020-02-06 16:38:25,181Z INFO
 [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
 (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
 IsVmDuringInitiatingVDSCommand, return: false, log id: 2107f52a
 2020-02-06 16:38:25,298Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
 (EE-ManagedThreadFactory-engine-Thread-216) [] Running command:
 RunVmCommand internal: false. Entities affected :  ID:
 df9dbac4-35c0-40ee-acd4-a1cfc959aa8b Type: VMAction group RUN_VM 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Stephen Panicho]

I used the cockpit-based hc setup and "option rpc-auth-allow-insecure" is
absent from /etc/glusterfs/glusterd.vol.

I'm going to redo the cluster this week and report back. Thanks for the tip!

On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic 
wrote:

> The hosts will still mount the volume via FUSE, but you might double check
> you set the storage up as Gluster and not NFS.
>
> Then gluster used to need some config in glusterd.vol to set
>
> option rpc-auth-allow-insecure on
>
> I’m not sure if that got added to a hyper converged setup or not, but I’d
> check it.
>
> On Feb 10, 2020, at 4:41 PM, Stephen Panicho  wrote:
>
> No, this was a relatively new cluster-- only a couple days old. Just a
> handful of VMs including the engine.
>
> On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:
>
>> Curious do the vms have active snapshots?
>>
>> On Mon, Feb 10, 2020 at 5:59 PM  wrote:
>>
>>> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running
>>> on CentOS 7.7 hosts. I was investigating poor Gluster performance and heard
>>> about libgfapi, so I thought I'd give it a shot. Looking through the
>>> documentation, followed by lots of threads and BZ reports, I've done the
>>> following to enable it:
>>>
>>> First, I shut down all VMs except the engine. Then...
>>>
>>> On the hosts:
>>> 1. setsebool -P virt_use_glusterfs on
>>> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>>>
>>> On the engine VM:
>>> 1. engine-config -s LibgfApiSupported=true --cver=4.3
>>> 2. systemctl restart ovirt-engine
>>>
>>> VMs now fail to launch. Am I doing this correctly? I should also note
>>> that the hosts still have the Gluster domain mounted via FUSE.
>>>
>>> Here's a relevant bit from engine.log:
>>>
>>> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
>>> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
>>> Could not read qcow2 header: Invalid argument.
>>>
>>> The full engine.log from one of the attempts:
>>>
>>> 2020-02-06 16:38:24,909Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (ForkJoinPool-1-worker-12) [] add VM
>>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
>>> 2020-02-06 16:38:25,010Z ERROR
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
>>> (ForkJoinPool-1-worker-12) [] Rerun VM
>>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
>>> node2.ovirt.trashnet.xyz'
>>> 2020-02-06 16:38:25,091Z WARN
>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID:
>>> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host
>>> node2.ovirt.trashnet.xyz.
>>> 2020-02-06 16:38:25,166Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object
>>> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]',
>>> sharedLocks=''}'
>>> 2020-02-06 16:38:25,179Z INFO
>>> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
>>> IsVmDuringInitiatingVDSCommand(
>>> IsVmDuringInitiatingVDSCommandParameters:{vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'}),
>>> log id: 2107f52a
>>> 2020-02-06 16:38:25,181Z INFO
>>> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
>>> IsVmDuringInitiatingVDSCommand, return: false, log id: 2107f52a
>>> 2020-02-06 16:38:25,298Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] Running command:
>>> RunVmCommand internal: false. Entities affected :  ID:
>>> df9dbac4-35c0-40ee-acd4-a1cfc959aa8b Type: VMAction group RUN_VM with role
>>> type USER
>>> 2020-02-06 16:38:25,313Z INFO
>>> [org.ovirt.engine.core.bll.utils.EmulatedMachineUtils]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] Emulated machine
>>> 'pc-q35-rhel7.6.0' which is different than that of the cluster is set for
>>> 'yumcache'(df9dbac4-35c0-40ee-acd4-a1cfc959aa8b)
>>> 2020-02-06 16:38:25,382Z INFO
>>> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
>>> UpdateVmDynamicDataVDSCommand(
>>> UpdateVmDynamicDataVDSCommandParameters:{hostId='null',
>>> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b',
>>> vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@9774a64'}),
>>> log id: 4a83911f
>>> 2020-02-06 16:38:25,417Z INFO
>>> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand]
>>> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
>>> UpdateVmDynamicDataVDSCommand, return: , log id: 4a83911f
>>> 2020-02-06 16:38:25,418Z INFO
>>> 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Darrell Budic]

The hosts will still mount the volume via FUSE, but you might double check you 
set the storage up as Gluster and not NFS.

Then gluster used to need some config in glusterd.vol to set 

option rpc-auth-allow-insecure on

I’m not sure if that got added to a hyper converged setup or not, but I’d check 
it.

> On Feb 10, 2020, at 4:41 PM, Stephen Panicho  wrote:
> 
> No, this was a relatively new cluster-- only a couple days old. Just a 
> handful of VMs including the engine.
> 
> On Mon, Feb 10, 2020 at 5:26 PM Jayme  > wrote:
> Curious do the vms have active snapshots?
> 
> On Mon, Feb 10, 2020 at 5:59 PM  > wrote:
> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running on 
> CentOS 7.7 hosts. I was investigating poor Gluster performance and heard 
> about libgfapi, so I thought I'd give it a shot. Looking through the 
> documentation, followed by lots of threads and BZ reports, I've done the 
> following to enable it:
> 
> First, I shut down all VMs except the engine. Then...
> 
> On the hosts:
> 1. setsebool -P virt_use_glusterfs on
> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
> 
> On the engine VM:
> 1. engine-config -s LibgfApiSupported=true --cver=4.3
> 2. systemctl restart ovirt-engine
> 
> VMs now fail to launch. Am I doing this correctly? I should also note that 
> the hosts still have the Gluster domain mounted via FUSE.
> 
> Here's a relevant bit from engine.log:
> 
> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive 
> file=gluster://node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native
>  
> :
>  Could not read qcow2 header: Invalid argument.
> 
> The full engine.log from one of the attempts:
> 
> 2020-02-06 16:38:24,909Z INFO  
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] 
> (ForkJoinPool-1-worker-12) [] add VM 
> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
> 2020-02-06 16:38:25,010Z ERROR 
> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] 
> (ForkJoinPool-1-worker-12) [] Rerun VM 
> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS 
> 'node2.ovirt.trashnet.xyz '
> 2020-02-06 16:38:25,091Z WARN  
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID: 
> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host 
> node2.ovirt.trashnet.xyz .
> 2020-02-06 16:38:25,166Z INFO  [org.ovirt.engine.core.bll.RunVmCommand] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object 
> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]', 
> sharedLocks=''}'
> 2020-02-06 16:38:25,179Z INFO  
> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] START, 
> IsVmDuringInitiatingVDSCommand( 
> IsVmDuringInitiatingVDSCommandParameters:{vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'}),
>  log id: 2107f52a
> 2020-02-06 16:38:25,181Z INFO  
> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH, 
> IsVmDuringInitiatingVDSCommand, return: false, log id: 2107f52a
> 2020-02-06 16:38:25,298Z INFO  [org.ovirt.engine.core.bll.RunVmCommand] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] Running command: RunVmCommand 
> internal: false. Entities affected :  ID: 
> df9dbac4-35c0-40ee-acd4-a1cfc959aa8b Type: VMAction group RUN_VM with role 
> type USER
> 2020-02-06 16:38:25,313Z INFO  
> [org.ovirt.engine.core.bll.utils.EmulatedMachineUtils] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] Emulated machine 
> 'pc-q35-rhel7.6.0' which is different than that of the cluster is set for 
> 'yumcache'(df9dbac4-35c0-40ee-acd4-a1cfc959aa8b)
> 2020-02-06 16:38:25,382Z INFO  
> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] 
> (EE-ManagedThreadFactory-engine-Thread-216) [] START, 
> UpdateVmDynamicDataVDSCommand( 
> UpdateVmDynamicDataVDSCommandParameters:{hostId='null', 
> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', 
> vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@9774a64'}),
>  log id: 4a83911f
> 2020-02-06 16:38:25,417Z INFO  
> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] 
> 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Stephen Panicho]

No, this was a relatively new cluster-- only a couple days old. Just a
handful of VMs including the engine.

On Mon, Feb 10, 2020 at 5:26 PM Jayme  wrote:

> Curious do the vms have active snapshots?
>
> On Mon, Feb 10, 2020 at 5:59 PM  wrote:
>
>> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running on
>> CentOS 7.7 hosts. I was investigating poor Gluster performance and heard
>> about libgfapi, so I thought I'd give it a shot. Looking through the
>> documentation, followed by lots of threads and BZ reports, I've done the
>> following to enable it:
>>
>> First, I shut down all VMs except the engine. Then...
>>
>> On the hosts:
>> 1. setsebool -P virt_use_glusterfs on
>> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>>
>> On the engine VM:
>> 1. engine-config -s LibgfApiSupported=true --cver=4.3
>> 2. systemctl restart ovirt-engine
>>
>> VMs now fail to launch. Am I doing this correctly? I should also note
>> that the hosts still have the Gluster domain mounted via FUSE.
>>
>> Here's a relevant bit from engine.log:
>>
>> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
>> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
>> Could not read qcow2 header: Invalid argument.
>>
>> The full engine.log from one of the attempts:
>>
>> 2020-02-06 16:38:24,909Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (ForkJoinPool-1-worker-12) [] add VM
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
>> 2020-02-06 16:38:25,010Z ERROR
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
>> (ForkJoinPool-1-worker-12) [] Rerun VM
>> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
>> node2.ovirt.trashnet.xyz'
>> 2020-02-06 16:38:25,091Z WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID:
>> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host
>> node2.ovirt.trashnet.xyz.
>> 2020-02-06 16:38:25,166Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object
>> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]',
>> sharedLocks=''}'
>> 2020-02-06 16:38:25,179Z INFO
>> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
>> IsVmDuringInitiatingVDSCommand(
>> IsVmDuringInitiatingVDSCommandParameters:{vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'}),
>> log id: 2107f52a
>> 2020-02-06 16:38:25,181Z INFO
>> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
>> IsVmDuringInitiatingVDSCommand, return: false, log id: 2107f52a
>> 2020-02-06 16:38:25,298Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] Running command:
>> RunVmCommand internal: false. Entities affected :  ID:
>> df9dbac4-35c0-40ee-acd4-a1cfc959aa8b Type: VMAction group RUN_VM with role
>> type USER
>> 2020-02-06 16:38:25,313Z INFO
>> [org.ovirt.engine.core.bll.utils.EmulatedMachineUtils]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] Emulated machine
>> 'pc-q35-rhel7.6.0' which is different than that of the cluster is set for
>> 'yumcache'(df9dbac4-35c0-40ee-acd4-a1cfc959aa8b)
>> 2020-02-06 16:38:25,382Z INFO
>> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
>> UpdateVmDynamicDataVDSCommand(
>> UpdateVmDynamicDataVDSCommandParameters:{hostId='null',
>> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b',
>> vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@9774a64'}),
>> log id: 4a83911f
>> 2020-02-06 16:38:25,417Z INFO
>> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
>> UpdateVmDynamicDataVDSCommand, return: , log id: 4a83911f
>> 2020-02-06 16:38:25,418Z INFO
>> [org.ovirt.engine.core.vdsbroker.CreateVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] START, CreateVDSCommand(
>> CreateVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452',
>> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', vm='VM [yumcache]'}), log id:
>> 5e07ba66
>> 2020-02-06 16:38:25,420Z INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
>> CreateBrokerVDSCommand(HostName = node1.ovirt.trashnet.xyz,
>> CreateVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452',
>> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', vm='VM [yumcache]'}), log id:
>> 1bfa03c4
>> 2020-02-06 16:38:25,424Z INFO
>> 

[ovirt-users] Re: Enabling Libgfapi in 4.3.8 - VMs won't start

[Jayme]

Curious do the vms have active snapshots?

On Mon, Feb 10, 2020 at 5:59 PM  wrote:

> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster running on
> CentOS 7.7 hosts. I was investigating poor Gluster performance and heard
> about libgfapi, so I thought I'd give it a shot. Looking through the
> documentation, followed by lots of threads and BZ reports, I've done the
> following to enable it:
>
> First, I shut down all VMs except the engine. Then...
>
> On the hosts:
> 1. setsebool -P virt_use_glusterfs on
> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf
>
> On the engine VM:
> 1. engine-config -s LibgfApiSupported=true --cver=4.3
> 2. systemctl restart ovirt-engine
>
> VMs now fail to launch. Am I doing this correctly? I should also note that
> the hosts still have the Gluster domain mounted via FUSE.
>
> Here's a relevant bit from engine.log:
>
> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster://
> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native:
> Could not read qcow2 header: Invalid argument.
>
> The full engine.log from one of the attempts:
>
> 2020-02-06 16:38:24,909Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-12) [] add VM
> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment
> 2020-02-06 16:38:25,010Z ERROR
> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
> (ForkJoinPool-1-worker-12) [] Rerun VM
> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS '
> node2.ovirt.trashnet.xyz'
> 2020-02-06 16:38:25,091Z WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID:
> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host
> node2.ovirt.trashnet.xyz.
> 2020-02-06 16:38:25,166Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object
> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]',
> sharedLocks=''}'
> 2020-02-06 16:38:25,179Z INFO
> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
> IsVmDuringInitiatingVDSCommand(
> IsVmDuringInitiatingVDSCommandParameters:{vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'}),
> log id: 2107f52a
> 2020-02-06 16:38:25,181Z INFO
> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
> IsVmDuringInitiatingVDSCommand, return: false, log id: 2107f52a
> 2020-02-06 16:38:25,298Z INFO  [org.ovirt.engine.core.bll.RunVmCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] Running command:
> RunVmCommand internal: false. Entities affected :  ID:
> df9dbac4-35c0-40ee-acd4-a1cfc959aa8b Type: VMAction group RUN_VM with role
> type USER
> 2020-02-06 16:38:25,313Z INFO
> [org.ovirt.engine.core.bll.utils.EmulatedMachineUtils]
> (EE-ManagedThreadFactory-engine-Thread-216) [] Emulated machine
> 'pc-q35-rhel7.6.0' which is different than that of the cluster is set for
> 'yumcache'(df9dbac4-35c0-40ee-acd4-a1cfc959aa8b)
> 2020-02-06 16:38:25,382Z INFO
> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
> UpdateVmDynamicDataVDSCommand(
> UpdateVmDynamicDataVDSCommandParameters:{hostId='null',
> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b',
> vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@9774a64'}),
> log id: 4a83911f
> 2020-02-06 16:38:25,417Z INFO
> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH,
> UpdateVmDynamicDataVDSCommand, return: , log id: 4a83911f
> 2020-02-06 16:38:25,418Z INFO
> [org.ovirt.engine.core.vdsbroker.CreateVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] START, CreateVDSCommand(
> CreateVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452',
> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', vm='VM [yumcache]'}), log id:
> 5e07ba66
> 2020-02-06 16:38:25,420Z INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-216) [] START,
> CreateBrokerVDSCommand(HostName = node1.ovirt.trashnet.xyz,
> CreateVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452',
> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', vm='VM [yumcache]'}), log id:
> 1bfa03c4
> 2020-02-06 16:38:25,424Z INFO
> [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuildUtils]
> (EE-ManagedThreadFactory-engine-Thread-216) [] Kernel FIPS - Guid:
> c3465ca2-395e-4c0c-b72e-b5b7153df452 fips: false
> 2020-02-06 16:38:25,435Z INFO
>