Re: [ovirt-users] Hosted engine on gluster problem

[David Gossage]

> >>
>>>>>> >>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Nir
>>>>>> >>>
>>>>>> >>> >
>>>>>> >>> > Could you add this feature to the next release of 3.6 branch?
>>>>>> >>> >
>>>>>> >>> > Thanks
>>>>>> >>> > Luiz
>>>>>> >>> >
>>>>>> >>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <
>>>>>> sbona...@redhat.com>
>>>>>> >>> > escreveu:
>>>>>> >>> >>
>>>>>> >>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <
>>>>>> db...@nrggos.com.au>
>>>>>> >>> >> wrote:
>>>>>> >>> >>>
>>>>>> >>> >>> My setup is hyperconverged. I have placed my test results in
>>>>>> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>>>>> >>> >>>
>>>>>> >>> >>
>>>>>> >>> >> Ok, so you're aware about the limitation of the single point of
>>>>>> >>> >> failure.
>>>>>> >>> >> If you drop the host referenced in hosted engine configuration
>>>>>> for the
>>>>>> >>> >> initial setup it won't be able to connect to shared storage
>>>>>> even if
>>>>>> >>> >> the
>>>>>> >>> >> other hosts in the cluster are up since the entry point is
>>>>>> down.
>>>>>> >>> >> Note that hyperconverged deployment is not supported in 3.6.
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>>
>>>>>> >>> >>>
>>>>>> >>> >>> Short description of setup:
>>>>>> >>> >>>
>>>>>> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3
>>>>>> across the 6
>>>>>> >>> >>> disks volume name hosted-engine.
>>>>>> >>> >>>
>>>>>> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to
>>>>>> the
>>>>>> >>> >>> host1.
>>>>>> >>> >>>
>>>>>> >>> >>> Installed hosted engine on host1 with the hosted engine
>>>>>> storage path
>>>>>> >>> >>> =
>>>>>> >>> >>> hosted-storage:/hosted-engine
>>>>>> >>> >>>
>>>>>> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added
>>>>>> to the
>>>>>> >>> >>> hosted engine. All works fine.
>>>>>> >>> >>>
>>>>>> >>> >>> Additional storage and non-hosted engine hosts added etc.
>>>>>> >>> >>>
>>>>>> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports
>>>>>> VM and
>>>>>> >>> >>> Cinder VM). Additional VM's are hosted by other storage -
>>>>>> cinder and
>>>>>> >>> >>> NFS.
>>>>>> >>> >>>
>>>>>> >>> >>> The system is in production.
>>>>>> >>> >>>
>>>>>> >>> >>>
>>>>>> >>> >>> Engine can be migrated around with the web interface.
>>>>>> >>> >>>
>>>>>> >>> >>>
>>>>>> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
>>>>>> >>> >>> upgraded
>>>>>> >>> >>> first , new Centos kernel requires host reboot.
>>>>>> >>> >>>
>>>>>> >>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade
>>>>

Re: [ovirt-users] Hosted engine on gluster problem

[Simone Tiraboschi]

t;> >>> >>> My setup is hyperconverged. I have placed my test results in
>>>>> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>>>> >>> >>>
>>>>> >>> >>
>>>>> >>> >> Ok, so you're aware about the limitation of the single point of
>>>>> >>> >> failure.
>>>>> >>> >> If you drop the host referenced in hosted engine configuration
>>>>> for the
>>>>> >>> >> initial setup it won't be able to connect to shared storage
>>>>> even if
>>>>> >>> >> the
>>>>> >>> >> other hosts in the cluster are up since the entry point is down.
>>>>> >>> >> Note that hyperconverged deployment is not supported in 3.6.
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>>
>>>>> >>> >>>
>>>>> >>> >>> Short description of setup:
>>>>> >>> >>>
>>>>> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across
>>>>> the 6
>>>>> >>> >>> disks volume name hosted-engine.
>>>>> >>> >>>
>>>>> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to
>>>>> the
>>>>> >>> >>> host1.
>>>>> >>> >>>
>>>>> >>> >>> Installed hosted engine on host1 with the hosted engine
>>>>> storage path
>>>>> >>> >>> =
>>>>> >>> >>> hosted-storage:/hosted-engine
>>>>> >>> >>>
>>>>> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added
>>>>> to the
>>>>> >>> >>> hosted engine. All works fine.
>>>>> >>> >>>
>>>>> >>> >>> Additional storage and non-hosted engine hosts added etc.
>>>>> >>> >>>
>>>>> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports
>>>>> VM and
>>>>> >>> >>> Cinder VM). Additional VM's are hosted by other storage -
>>>>> cinder and
>>>>> >>> >>> NFS.
>>>>> >>> >>>
>>>>> >>> >>> The system is in production.
>>>>> >>> >>>
>>>>> >>> >>>
>>>>> >>> >>> Engine can be migrated around with the web interface.
>>>>> >>> >>>
>>>>> >>> >>>
>>>>> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
>>>>> >>> >>> upgraded
>>>>> >>> >>> first , new Centos kernel requires host reboot.
>>>>> >>> >>>
>>>>> >>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade
>>>>> and
>>>>> >>> >>> Reboot
>>>>> >>> >>> h3 - No issues - Local maintenance removed from h3.
>>>>> >>> >>>
>>>>> >>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade
>>>>> and
>>>>> >>> >>> Reboot
>>>>> >>> >>> h2 - No issues - Local maintenance removed from h2.
>>>>> >>> >>>
>>>>> >>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and
>>>>> reboot
>>>>> >>> >>> h1 -
>>>>> >>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3
>>>>> on
>>>>> >>> >>> same
>>>>> >>> >>> gluster volume pauses.
>>>>> >>> >>>
>>>>> >>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with
>>>>> all
>>>>> >>&

Re: [ovirt-users] Hosted engine on gluster problem

[David Gossage]

ould be able to connect to
>>> >>> gluster
>>> >>> storage domain even if a brick is down.
>>> >>>
>>> >>> If you don't have this option in engine , you probably cannot add it
>>> with
>>> >>> hosted
>>> >>> engine setup, since for editing it you must put the storage domain in
>>> >>> maintenance
>>> >>> and if you do this the engine vm will be killed :-) This is is one of
>>> >>> the issues with
>>> >>> engine managing the storage domain it runs on.
>>> >>>
>>> >>> I think the best way to avoid this issue, is to add a DNS entry
>>> >>> providing the addresses
>>> >>> of all the gluster bricks, and use this address for the gluster
>>> >>> storage domain. This way
>>> >>> the glusterfs mount helper can mount the domain even if one of the
>>> >>> gluster bricks
>>> >>> are down.
>>> >>>
>>> >>> Again, we will need some magic from the hosted engine developers to
>>> >>> modify the
>>> >>> address of the hosted engine gluster domain on existing system.
>>> >>
>>> >>
>>> >> Magic won't happen without a bz :-) please open one describing what's
>>> >> requested.
>>> >>
>>> >>
>>> >>>
>>> >>>
>>> >>> Nir
>>> >>>
>>> >>> >
>>> >>> > Could you add this feature to the next release of 3.6 branch?
>>> >>> >
>>> >>> > Thanks
>>> >>> > Luiz
>>> >>> >
>>> >>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <
>>> sbona...@redhat.com>
>>> >>> > escreveu:
>>> >>> >>
>>> >>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <
>>> db...@nrggos.com.au>
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> My setup is hyperconverged. I have placed my test results in
>>> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>> >>> >>>
>>> >>> >>
>>> >>> >> Ok, so you're aware about the limitation of the single point of
>>> >>> >> failure.
>>> >>> >> If you drop the host referenced in hosted engine configuration
>>> for the
>>> >>> >> initial setup it won't be able to connect to shared storage even
>>> if
>>> >>> >> the
>>> >>> >> other hosts in the cluster are up since the entry point is down.
>>> >>> >> Note that hyperconverged deployment is not supported in 3.6.
>>> >>> >>
>>> >>> >>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> Short description of setup:
>>> >>> >>>
>>> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across
>>> the 6
>>> >>> >>> disks volume name hosted-engine.
>>> >>> >>>
>>> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
>>> >>> >>> host1.
>>> >>> >>>
>>> >>> >>> Installed hosted engine on host1 with the hosted engine storage
>>> path
>>> >>> >>> =
>>> >>> >>> hosted-storage:/hosted-engine
>>> >>> >>>
>>> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to
>>> the
>>> >>> >>> hosted engine. All works fine.
>>> >>> >>>
>>> >>> >>> Additional storage and non-hosted engine hosts added etc.
>>> >>> >>>
>>> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM
>>> and
>>> >>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder
>>> and
>>> >>> >>> NFS.
>>> >>> >>>
>>> >>> >>> The system is in production.
>>> >>> >>>
>>> >>>

Re: [ovirt-users] Hosted engine on gluster problem

[Luiz Claudio Prazeres Goncalves]

gt;> >>> are down.
>> >>>
>> >>> Again, we will need some magic from the hosted engine developers to
>> >>> modify the
>> >>> address of the hosted engine gluster domain on existing system.
>> >>
>> >>
>> >> Magic won't happen without a bz :-) please open one describing what's
>> >> requested.
>> >>
>> >>
>> >>>
>> >>>
>> >>> Nir
>> >>>
>> >>> >
>> >>> > Could you add this feature to the next release of 3.6 branch?
>> >>> >
>> >>> > Thanks
>> >>> > Luiz
>> >>> >
>> >>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <
>> sbona...@redhat.com>
>> >>> > escreveu:
>> >>> >>
>> >>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <
>> db...@nrggos.com.au>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> My setup is hyperconverged. I have placed my test results in
>> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>> >>> >>>
>> >>> >>
>> >>> >> Ok, so you're aware about the limitation of the single point of
>> >>> >> failure.
>> >>> >> If you drop the host referenced in hosted engine configuration for
>> the
>> >>> >> initial setup it won't be able to connect to shared storage even if
>> >>> >> the
>> >>> >> other hosts in the cluster are up since the entry point is down.
>> >>> >> Note that hyperconverged deployment is not supported in 3.6.
>> >>> >>
>> >>> >>
>> >>> >>>
>> >>> >>>
>> >>> >>> Short description of setup:
>> >>> >>>
>> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across
>> the 6
>> >>> >>> disks volume name hosted-engine.
>> >>> >>>
>> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
>> >>> >>> host1.
>> >>> >>>
>> >>> >>> Installed hosted engine on host1 with the hosted engine storage
>> path
>> >>> >>> =
>> >>> >>> hosted-storage:/hosted-engine
>> >>> >>>
>> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to
>> the
>> >>> >>> hosted engine. All works fine.
>> >>> >>>
>> >>> >>> Additional storage and non-hosted engine hosts added etc.
>> >>> >>>
>> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM
>> and
>> >>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder
>> and
>> >>> >>> NFS.
>> >>> >>>
>> >>> >>> The system is in production.
>> >>> >>>
>> >>> >>>
>> >>> >>> Engine can be migrated around with the web interface.
>> >>> >>>
>> >>> >>>
>> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
>> >>> >>> upgraded
>> >>> >>> first , new Centos kernel requires host reboot.
>> >>> >>>
>> >>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and
>> >>> >>> Reboot
>> >>> >>> h3 - No issues - Local maintenance removed from h3.
>> >>> >>>
>> >>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and
>> >>> >>> Reboot
>> >>> >>> h2 - No issues - Local maintenance removed from h2.
>> >>> >>>
>> >>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and
>> reboot
>> >>> >>> h1 -
>> >>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on
>> >>> >>> same
>> >>> >>> gluster volume pauses.
>> >>> >>>
>> >>> >>> - Host 1 takes ab

Re: [ovirt-users] Hosted engine on gluster problem

[Sandro Bonazzola]

> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <db...@nrggos.com.au
> >
> >>> >> wrote:
> >>> >>>
> >>> >>> My setup is hyperconverged. I have placed my test results in
> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
> >>> >>>
> >>> >>
> >>> >> Ok, so you're aware about the limitation of the single point of
> >>> >> failure.
> >>> >> If you drop the host referenced in hosted engine configuration for
> the
> >>> >> initial setup it won't be able to connect to shared storage even if
> >>> >> the
> >>> >> other hosts in the cluster are up since the entry point is down.
> >>> >> Note that hyperconverged deployment is not supported in 3.6.
> >>> >>
> >>> >>
> >>> >>>
> >>> >>>
> >>> >>> Short description of setup:
> >>> >>>
> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across the
> 6
> >>> >>> disks volume name hosted-engine.
> >>> >>>
> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
> >>> >>> host1.
> >>> >>>
> >>> >>> Installed hosted engine on host1 with the hosted engine storage
> path
> >>> >>> =
> >>> >>> hosted-storage:/hosted-engine
> >>> >>>
> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to the
> >>> >>> hosted engine. All works fine.
> >>> >>>
> >>> >>> Additional storage and non-hosted engine hosts added etc.
> >>> >>>
> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
> >>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder
> and
> >>> >>> NFS.
> >>> >>>
> >>> >>> The system is in production.
> >>> >>>
> >>> >>>
> >>> >>> Engine can be migrated around with the web interface.
> >>> >>>
> >>> >>>
> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
> >>> >>> upgraded
> >>> >>> first , new Centos kernel requires host reboot.
> >>> >>>
> >>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and
> >>> >>> Reboot
> >>> >>> h3 - No issues - Local maintenance removed from h3.
> >>> >>>
> >>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and
> >>> >>> Reboot
> >>> >>> h2 - No issues - Local maintenance removed from h2.
> >>> >>>
> >>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and
> reboot
> >>> >>> h1 -
> >>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on
> >>> >>> same
> >>> >>> gluster volume pauses.
> >>> >>>
> >>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all
> >>> >>> it's
> >>> >>> normal BIOS probing)
> >>> >>>
> >>> >>> - Engine starts after h1 comes back and stabilises
> >>> >>>
> >>> >>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
> >>> >>> time.
> >>> >>> I can do no diagnosis on the 2 VMs as the engine is not available.
> >>> >>>
> >>> >>> - Local maintenance removed from h1
> >>> >>>
> >>> >>>
> >>> >>> I don't believe the issue is with gluster itself as the volume
> >>> >>> remains
> >>> >>> accessible on all hosts during this time albeit with a missing
> server
> >>> >>> (gluster volume status) as each gluster server is rebooted.
> >>> >>>
> >>> >>> Gluster was upgraded as part of the process, no issues were seen
> >>> >>> here.
> >>> >>>
> >>> >>>
> >>> >>> I have been able

Re: [ovirt-users] Hosted engine on gluster problem

[Nir Soffer]

 other hosts in the cluster are up since the entry point is down.
>>> >> Note that hyperconverged deployment is not supported in 3.6.
>>> >>
>>> >>
>>> >>>
>>> >>>
>>> >>> Short description of setup:
>>> >>>
>>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6
>>> >>> disks volume name hosted-engine.
>>> >>>
>>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
>>> >>> host1.
>>> >>>
>>> >>> Installed hosted engine on host1 with the hosted engine storage path
>>> >>> =
>>> >>> hosted-storage:/hosted-engine
>>> >>>
>>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to the
>>> >>> hosted engine. All works fine.
>>> >>>
>>> >>> Additional storage and non-hosted engine hosts added etc.
>>> >>>
>>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
>>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder and
>>> >>> NFS.
>>> >>>
>>> >>> The system is in production.
>>> >>>
>>> >>>
>>> >>> Engine can be migrated around with the web interface.
>>> >>>
>>> >>>
>>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
>>> >>> upgraded
>>> >>> first , new Centos kernel requires host reboot.
>>> >>>
>>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and
>>> >>> Reboot
>>> >>> h3 - No issues - Local maintenance removed from h3.
>>> >>>
>>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and
>>> >>> Reboot
>>> >>> h2 - No issues - Local maintenance removed from h2.
>>> >>>
>>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot
>>> >>> h1 -
>>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on
>>> >>> same
>>> >>> gluster volume pauses.
>>> >>>
>>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all
>>> >>> it's
>>> >>> normal BIOS probing)
>>> >>>
>>> >>> - Engine starts after h1 comes back and stabilises
>>> >>>
>>> >>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
>>> >>> time.
>>> >>> I can do no diagnosis on the 2 VMs as the engine is not available.
>>> >>>
>>> >>> - Local maintenance removed from h1
>>> >>>
>>> >>>
>>> >>> I don't believe the issue is with gluster itself as the volume
>>> >>> remains
>>> >>> accessible on all hosts during this time albeit with a missing server
>>> >>> (gluster volume status) as each gluster server is rebooted.
>>> >>>
>>> >>> Gluster was upgraded as part of the process, no issues were seen
>>> >>> here.
>>> >>>
>>> >>>
>>> >>> I have been able to duplicate the issue without the upgrade by
>>> >>> following
>>> >>> the same sort of timeline.
>>> >>>
>>> >>>
>>> >>> 
>>> >>> From: Sandro Bonazzola <sbona...@redhat.com>
>>> >>> Sent: Monday, 11 April 2016 7:11 PM
>>> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak;
>>> >>> Sahina
>>> >>> Bose
>>> >>> Cc: Bond, Darryl; users
>>> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
>>> >>> <h...@tbi.univie.ac.at<mailto:h...@tbi.univie.ac.at>> wrote:
>>> >>> Hi Darryl,
>>> >>>
>>> >>> I'm still experimenting with my oVirt installation so I tried to
>>> >>> recreate the problems you've described.
>>&

Re: [ovirt-users] Hosted engine on gluster problem

[Luiz Claudio Prazeres Goncalves]

etup it won't be able to connect to shared storage even if
>>> the
>>> >> other hosts in the cluster are up since the entry point is down.
>>> >> Note that hyperconverged deployment is not supported in 3.6.
>>> >>
>>> >>
>>> >>>
>>> >>>
>>> >>> Short description of setup:
>>> >>>
>>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6
>>> >>> disks volume name hosted-engine.
>>> >>>
>>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
>>> host1.
>>> >>>
>>> >>> Installed hosted engine on host1 with the hosted engine storage path
>>> =
>>> >>> hosted-storage:/hosted-engine
>>> >>>
>>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to the
>>> >>> hosted engine. All works fine.
>>> >>>
>>> >>> Additional storage and non-hosted engine hosts added etc.
>>> >>>
>>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
>>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder and
>>> NFS.
>>> >>>
>>> >>> The system is in production.
>>> >>>
>>> >>>
>>> >>> Engine can be migrated around with the web interface.
>>> >>>
>>> >>>
>>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
>>> upgraded
>>> >>> first , new Centos kernel requires host reboot.
>>> >>>
>>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and
>>> Reboot
>>> >>> h3 - No issues - Local maintenance removed from h3.
>>> >>>
>>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and
>>> Reboot
>>> >>> h2 - No issues - Local maintenance removed from h2.
>>> >>>
>>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot
>>> h1 -
>>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on
>>> same
>>> >>> gluster volume pauses.
>>> >>>
>>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all
>>> it's
>>> >>> normal BIOS probing)
>>> >>>
>>> >>> - Engine starts after h1 comes back and stabilises
>>> >>>
>>> >>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
>>> time.
>>> >>> I can do no diagnosis on the 2 VMs as the engine is not available.
>>> >>>
>>> >>> - Local maintenance removed from h1
>>> >>>
>>> >>>
>>> >>> I don't believe the issue is with gluster itself as the volume
>>> remains
>>> >>> accessible on all hosts during this time albeit with a missing server
>>> >>> (gluster volume status) as each gluster server is rebooted.
>>> >>>
>>> >>> Gluster was upgraded as part of the process, no issues were seen
>>> here.
>>> >>>
>>> >>>
>>> >>> I have been able to duplicate the issue without the upgrade by
>>> following
>>> >>> the same sort of timeline.
>>> >>>
>>> >>>
>>> >>> 
>>> >>> From: Sandro Bonazzola <sbona...@redhat.com>
>>> >>> Sent: Monday, 11 April 2016 7:11 PM
>>> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak;
>>> Sahina
>>> >>> Bose
>>> >>> Cc: Bond, Darryl; users
>>> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
>>> >>> <h...@tbi.univie.ac.at<mailto:h...@tbi.univie.ac.at>> wrote:
>>> >>> Hi Darryl,
>>> >>>
>>> >>> I'm still experimenting with my oVirt installation so I tried to
>>> >>> recreate the problems you've described.
>>> >>>
>>> >>> My setup has three HA hosts for virtualization and three

Re: [ovirt-users] Hosted engine on gluster problem

[Sahina Bose]

de and
reboot h1 - engine crashes and does not start elsewhere,
VM(cinder)  on h3 on same gluster volume pauses.

- Host 1 takes about 5 minutes to reboot (Enterprise box with all
it's normal BIOS probing)

- Engine starts after h1 comes back and stabilises

- VM(cinder) unpauses itself,  VM(reports) continued fine the
whole time. I can do no diagnosis on the 2 VMs as the engine is
not available.

- Local maintenance removed from h​1


I don't believe the issue is with gluster itself as the volume
remains accessible on all hosts during this time albeit with a
missing server (gluster volume status) as each gluster server is
rebooted.

Gluster was upgraded as part of the process, no issues were seen here.


I have been able to duplicate the issue without the upgrade by
following the same sort of timeline.



From: Sandro Bonazzola <sbona...@redhat.com
<mailto:sbona...@redhat.com>>
Sent: Monday, 11 April 2016 7:11 PM
To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak;
Sahina Bose
Cc: Bond, Darryl; users
Subject: Re: [ovirt-users] Hosted engine on gluster problem



On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
<h...@tbi.univie.ac.at
<mailto:h...@tbi.univie.ac.at><mailto:h...@tbi.univie.ac.at
<mailto:h...@tbi.univie.ac.at>>> wrote:
Hi Darryl,

I'm still experimenting with my oVirt installation so I tried to
recreate the problems you've described.

My setup has three HA hosts for virtualization and three machines
for the gluster replica 3 setup.

I manually migrated the Engine from the initial install host (one)
to host three. Then shut down host one manually and interrupted the
fencing mechanisms so the host stayed down. This didn't bother the
Engine VM at all.

Did you move the host one to maintenance before shutting down?
Or is this a crash recovery test?



To make things a bit more challenging I then shut down host three
while running the Engine VM. Of course the Engine was down for some
time until host two detected the problem. It started the Engine VM
and everything seems to be running quite well without the initial
install host.

Thanks for the feedback!



My only problem is that the HA agent on host two and three refuse to
start after a reboot due to the fact that the configuration of the
hosted engine is missing. I wrote another mail to users@ovirt.org
<mailto:users@ovirt.org><mailto:users@ovirt.org
<mailto:users@ovirt.org>>
about that.

This is weird. Martin,  Simone can you please investigate on this?




Cheers
Richard

On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> There seems to be a pretty severe bug with using hosted engine
on gluster.
>
> If the host that was used as the initial hosted-engine --deploy
host goes away, the engine VM wil crash and cannot be restarted
until the host comes back.

is this an Hyperconverged setup?


>
> This is regardless of which host the engine was currently running.
>
>
> The issue seems to be buried in the bowels of VDSM and is not an
issue with gluster itself.

Sahina, can you please investigate on this?


>
> The gluster filesystem is still accessable from the host that
was running the engine. The issue has been submitted to bugzilla
but the fix is some way off (4.1).
>
>
> Can my hosted engine be converted to use NFS (using the gluster
NFS server on the same filesystem) without rebuilding my hosted
engine (ie change domainType=glusterfs to domainType=nfs)?

>
> What effect would that have on the hosted-engine storage domain
inside oVirt, ie would the same filesystem be mounted twice or
would it just break.
>
>
> Will this actually fix the problem, does it have the same issue
when the hosted engine is on NFS?
>
>
> Darryl
>
>
>
>
> 
>
> The contents of this electronic message and any attachments are
intended only for the addressee and may contain legally
privileged, personal, sensitive or confidential information. If
you are not the intended addressee, and have received this email,
any transmission, distribution, downloading, printing or
photocopying of the contents of this message or attachments is
strictly prohibited. Any legal privilege or confidentiality
attached to this message and attachments is not waived, lost or
destroyed by reason of delivery to any person other than intended
addressee. If you have received this message and are not the
intended addressee you should no

Re: [ovirt-users] Hosted engine on gluster problem

[Sandro Bonazzola]

olume pauses.
> >>>
> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all it's
> >>> normal BIOS probing)
> >>>
> >>> - Engine starts after h1 comes back and stabilises
> >>>
> >>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
> time.
> >>> I can do no diagnosis on the 2 VMs as the engine is not available.
> >>>
> >>> - Local maintenance removed from h1
> >>>
> >>>
> >>> I don't believe the issue is with gluster itself as the volume remains
> >>> accessible on all hosts during this time albeit with a missing server
> >>> (gluster volume status) as each gluster server is rebooted.
> >>>
> >>> Gluster was upgraded as part of the process, no issues were seen here.
> >>>
> >>>
> >>> I have been able to duplicate the issue without the upgrade by
> following
> >>> the same sort of timeline.
> >>>
> >>>
> >>> 
> >>> From: Sandro Bonazzola <sbona...@redhat.com>
> >>> Sent: Monday, 11 April 2016 7:11 PM
> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak;
> Sahina
> >>> Bose
> >>> Cc: Bond, Darryl; users
> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
> >>>
> >>>
> >>>
> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
> >>> <h...@tbi.univie.ac.at<mailto:h...@tbi.univie.ac.at>> wrote:
> >>> Hi Darryl,
> >>>
> >>> I'm still experimenting with my oVirt installation so I tried to
> >>> recreate the problems you've described.
> >>>
> >>> My setup has three HA hosts for virtualization and three machines
> >>> for the gluster replica 3 setup.
> >>>
> >>> I manually migrated the Engine from the initial install host (one)
> >>> to host three. Then shut down host one manually and interrupted the
> >>> fencing mechanisms so the host stayed down. This didn't bother the
> >>> Engine VM at all.
> >>>
> >>> Did you move the host one to maintenance before shutting down?
> >>> Or is this a crash recovery test?
> >>>
> >>>
> >>>
> >>> To make things a bit more challenging I then shut down host three
> >>> while running the Engine VM. Of course the Engine was down for some
> >>> time until host two detected the problem. It started the Engine VM
> >>> and everything seems to be running quite well without the initial
> >>> install host.
> >>>
> >>> Thanks for the feedback!
> >>>
> >>>
> >>>
> >>> My only problem is that the HA agent on host two and three refuse to
> >>> start after a reboot due to the fact that the configuration of the
> >>> hosted engine is missing. I wrote another mail to
> >>> users@ovirt.org<mailto:users@ovirt.org>
> >>> about that.
> >>>
> >>> This is weird. Martin,  Simone can you please investigate on this?
> >>>
> >>>
> >>>
> >>>
> >>> Cheers
> >>> Richard
> >>>
> >>> On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> >>> > There seems to be a pretty severe bug with using hosted engine on
> >>> > gluster.
> >>> >
> >>> > If the host that was used as the initial hosted-engine --deploy host
> >>> > goes away, the engine VM wil crash and cannot be restarted until the
> host
> >>> > comes back.
> >>>
> >>> is this an Hyperconverged setup?
> >>>
> >>>
> >>> >
> >>> > This is regardless of which host the engine was currently running.
> >>> >
> >>> >
> >>> > The issue seems to be buried in the bowels of VDSM and is not an
> issue
> >>> > with gluster itself.
> >>>
> >>> Sahina, can you please investigate on this?
> >>>
> >>>
> >>> >
> >>> > The gluster filesystem is still accessable from the host that was
> >>> > running the engine. The issue has been submitted to bugzilla but the
> fix is
> >>> > some way off (4.1).
> >>> >
> >>> >
> >>> > Can my hosted engine be converted to use NFS (using the gluster NFS
> &

Re: [ovirt-users] Hosted engine on gluster problem

[Sandro Bonazzola]

On Tue, Apr 12, 2016 at 2:05 PM, Luiz Claudio Prazeres Goncalves <
luiz...@gmail.com> wrote:

> Hi Sandro, I've been using gluster with 3 external hosts for a while and
> things are working pretty well, however this single point of failure looks
> like a simple feature to implement,but critical to anyone who wants to use
> gluster on production  . This is not hyperconvergency which has other
> issues/implications. So , why not have this feature out on 3.6 branch? It
> looks like just let vdsm use the 'backupvol-server' option when mounting
> the engine domain and make the property tests.
>
> Could you add this feature to the next release of 3.6 branch?
>

Having it in 3.6 is hard in terms of integration team capacity. Also
consider than 4.0 will be probably out before we'll be able to backport it
to 3.6.



>
> Thanks
> Luiz
>
> Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <sbona...@redhat.com>
> escreveu:
>
>> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <db...@nrggos.com.au>
>> wrote:
>>
>>> My setup is hyperconverged. I have placed my test results in
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>>
>>>
>> Ok, so you're aware about the limitation of the single point of failure.
>> If you drop the host referenced in hosted engine configuration for the
>> initial setup it won't be able to connect to shared storage even if the
>> other hosts in the cluster are up since the entry point is down.
>> Note that hyperconverged deployment is not supported in 3.6.
>>
>>
>>
>>>
>>> Short description of setup:
>>>
>>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6
>>> disks volume name hosted-engine.
>>>
>>> Hostname hosted-storage configured in /etc//hosts to point to the host1.
>>>
>>> Installed hosted engine on host1 with the hosted engine storage path =
>>> hosted-storage:/hosted-engine
>>>
>>> Install first engine on h1 successful. Hosts h2 and h3 added to the
>>> hosted engine. All works fine.
>>>
>>> Additional storage and non-hosted engine hosts added etc.
>>>
>>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
>>> Cinder VM). Additional VM's are hosted by other storage - cinder and NFS.
>>>
>>> The system is in production.
>>>
>>>
>>> Engine can be migrated around with the web interface.
>>>
>>>
>>> - 3.6.4 upgrade released, follow the upgrade guide, engine is upgraded
>>> first , new Centos kernel requires host reboot.
>>>
>>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and Reboot
>>> h3 - No issues - Local maintenance removed from h3.
>>>
>>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and Reboot
>>> h2 - No issues - Local maintenance removed from h2.
>>>
>>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot h1
>>> - engine crashes and does not start elsewhere, VM(cinder)  on h3 on same
>>> gluster volume pauses.
>>>
>>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all it's
>>> normal BIOS probing)
>>>
>>> - Engine starts after h1 comes back and stabilises
>>>
>>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
>>> time. I can do no diagnosis on the 2 VMs as the engine is not available.
>>>
>>> - Local maintenance removed from h​1
>>>
>>>
>>> I don't believe the issue is with gluster itself as the volume remains
>>> accessible on all hosts during this time albeit with a missing server
>>> (gluster volume status) as each gluster server is rebooted.
>>>
>>> Gluster was upgraded as part of the process, no issues were seen here.
>>>
>>>
>>> I have been able to duplicate the issue without the upgrade by following
>>> the same sort of timeline.
>>>
>>>
>>> 
>>> From: Sandro Bonazzola <sbona...@redhat.com>
>>> Sent: Monday, 11 April 2016 7:11 PM
>>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak; Sahina
>>> Bose
>>> Cc: Bond, Darryl; users
>>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
>>>
>>>
>>>
>>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck <h...@tbi.univie.ac.at
>>> <mailto:h...@tbi.univie.ac.at>> wrote:
>>> Hi Darryl,
>>>
>>> I'm still e

Re: [ovirt-users] Hosted engine on gluster problem

[Nir Soffer]

On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves
<luiz...@gmail.com> wrote:
> Hi Sandro, I've been using gluster with 3 external hosts for a while and
> things are working pretty well, however this single point of failure looks
> like a simple feature to implement,but critical to anyone who wants to use
> gluster on production  . This is not hyperconvergency which has other
> issues/implications. So , why not have this feature out on 3.6 branch? It
> looks like just let vdsm use the 'backupvol-server' option when mounting the
> engine domain and make the property tests.

Can you explain what is the problem, and what is the suggested solution?

Engine and vdsm already support the backupvol-server option - you can
define this option in the storage domain options when you create a gluster
storage domain. With this option vdsm should be able to connect to gluster
storage domain even if a brick is down.

If you don't have this option in engine , you probably cannot add it with hosted
engine setup, since for editing it you must put the storage domain in
maintenance
and if you do this the engine vm will be killed :-) This is is one of
the issues with
engine managing the storage domain it runs on.

I think the best way to avoid this issue, is to add a DNS entry
providing the addresses
of all the gluster bricks, and use this address for the gluster
storage domain. This way
the glusterfs mount helper can mount the domain even if one of the
gluster bricks
are down.

Again, we will need some magic from the hosted engine developers to modify the
address of the hosted engine gluster domain on existing system.

Nir

>
> Could you add this feature to the next release of 3.6 branch?
>
> Thanks
> Luiz
>
> Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <sbona...@redhat.com>
> escreveu:
>>
>> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <db...@nrggos.com.au>
>> wrote:
>>>
>>> My setup is hyperconverged. I have placed my test results in
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>>
>>
>> Ok, so you're aware about the limitation of the single point of failure.
>> If you drop the host referenced in hosted engine configuration for the
>> initial setup it won't be able to connect to shared storage even if the
>> other hosts in the cluster are up since the entry point is down.
>> Note that hyperconverged deployment is not supported in 3.6.
>>
>>
>>>
>>>
>>> Short description of setup:
>>>
>>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6
>>> disks volume name hosted-engine.
>>>
>>> Hostname hosted-storage configured in /etc//hosts to point to the host1.
>>>
>>> Installed hosted engine on host1 with the hosted engine storage path =
>>> hosted-storage:/hosted-engine
>>>
>>> Install first engine on h1 successful. Hosts h2 and h3 added to the
>>> hosted engine. All works fine.
>>>
>>> Additional storage and non-hosted engine hosts added etc.
>>>
>>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
>>> Cinder VM). Additional VM's are hosted by other storage - cinder and NFS.
>>>
>>> The system is in production.
>>>
>>>
>>> Engine can be migrated around with the web interface.
>>>
>>>
>>> - 3.6.4 upgrade released, follow the upgrade guide, engine is upgraded
>>> first , new Centos kernel requires host reboot.
>>>
>>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and Reboot
>>> h3 - No issues - Local maintenance removed from h3.
>>>
>>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and Reboot
>>> h2 - No issues - Local maintenance removed from h2.
>>>
>>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot h1 -
>>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on same
>>> gluster volume pauses.
>>>
>>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all it's
>>> normal BIOS probing)
>>>
>>> - Engine starts after h1 comes back and stabilises
>>>
>>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole time.
>>> I can do no diagnosis on the 2 VMs as the engine is not available.
>>>
>>> - Local maintenance removed from h1
>>>
>>>
>>> I don't believe the issue is with gluster itself as the volume remains
>>> accessible on all hosts during this time albeit with a missing server
>>> (gluster volume status) as each gluster server is rebooted.
&g

Re: [ovirt-users] Hosted engine on gluster problem

[Luiz Claudio Prazeres Goncalves]

Hi Sandro, I've been using gluster with 3 external hosts for a while and
things are working pretty well, however this single point of failure looks
like a simple feature to implement,but critical to anyone who wants to use
gluster on production  . This is not hyperconvergency which has other
issues/implications. So , why not have this feature out on 3.6 branch? It
looks like just let vdsm use the 'backupvol-server' option when mounting
the engine domain and make the property tests.

Could you add this feature to the next release of 3.6 branch?

Thanks
Luiz

Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <sbona...@redhat.com>
escreveu:

> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <db...@nrggos.com.au>
> wrote:
>
>> My setup is hyperconverged. I have placed my test results in
>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>
>>
> Ok, so you're aware about the limitation of the single point of failure.
> If you drop the host referenced in hosted engine configuration for the
> initial setup it won't be able to connect to shared storage even if the
> other hosts in the cluster are up since the entry point is down.
> Note that hyperconverged deployment is not supported in 3.6.
>
>
>
>>
>> Short description of setup:
>>
>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6
>> disks volume name hosted-engine.
>>
>> Hostname hosted-storage configured in /etc//hosts to point to the host1.
>>
>> Installed hosted engine on host1 with the hosted engine storage path =
>> hosted-storage:/hosted-engine
>>
>> Install first engine on h1 successful. Hosts h2 and h3 added to the
>> hosted engine. All works fine.
>>
>> Additional storage and non-hosted engine hosts added etc.
>>
>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
>> Cinder VM). Additional VM's are hosted by other storage - cinder and NFS.
>>
>> The system is in production.
>>
>>
>> Engine can be migrated around with the web interface.
>>
>>
>> - 3.6.4 upgrade released, follow the upgrade guide, engine is upgraded
>> first , new Centos kernel requires host reboot.
>>
>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and Reboot
>> h3 - No issues - Local maintenance removed from h3.
>>
>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and Reboot
>> h2 - No issues - Local maintenance removed from h2.
>>
>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot h1 -
>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on same
>> gluster volume pauses.
>>
>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all it's
>> normal BIOS probing)
>>
>> - Engine starts after h1 comes back and stabilises
>>
>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole time.
>> I can do no diagnosis on the 2 VMs as the engine is not available.
>>
>> - Local maintenance removed from h​1
>>
>>
>> I don't believe the issue is with gluster itself as the volume remains
>> accessible on all hosts during this time albeit with a missing server
>> (gluster volume status) as each gluster server is rebooted.
>>
>> Gluster was upgraded as part of the process, no issues were seen here.
>>
>>
>> I have been able to duplicate the issue without the upgrade by following
>> the same sort of timeline.
>>
>>
>> 
>> From: Sandro Bonazzola <sbona...@redhat.com>
>> Sent: Monday, 11 April 2016 7:11 PM
>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak; Sahina
>> Bose
>> Cc: Bond, Darryl; users
>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
>>
>>
>>
>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck <h...@tbi.univie.ac.at
>> <mailto:h...@tbi.univie.ac.at>> wrote:
>> Hi Darryl,
>>
>> I'm still experimenting with my oVirt installation so I tried to
>> recreate the problems you've described.
>>
>> My setup has three HA hosts for virtualization and three machines
>> for the gluster replica 3 setup.
>>
>> I manually migrated the Engine from the initial install host (one)
>> to host three. Then shut down host one manually and interrupted the
>> fencing mechanisms so the host stayed down. This didn't bother the
>> Engine VM at all.
>>
>> Did you move the host one to maintenance before shutting down?
>> Or is this a crash recovery test?
>>
>>
>>
>> To make things a bit more challenging 

Re: [ovirt-users] Hosted engine on gluster problem

[Sandro Bonazzola]

On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <db...@nrggos.com.au> wrote:

> My setup is hyperconverged. I have placed my test results in
> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>
>
Ok, so you're aware about the limitation of the single point of failure. If
you drop the host referenced in hosted engine configuration for the initial
setup it won't be able to connect to shared storage even if the other hosts
in the cluster are up since the entry point is down.
Note that hyperconverged deployment is not supported in 3.6.



>
> Short description of setup:
>
> 3 hosts with 2 disks each set up with gluster replica 3 across the 6 disks
> volume name hosted-engine.
>
> Hostname hosted-storage configured in /etc//hosts to point to the host1.
>
> Installed hosted engine on host1 with the hosted engine storage path =
> hosted-storage:/hosted-engine
>
> Install first engine on h1 successful. Hosts h2 and h3 added to the hosted
> engine. All works fine.
>
> Additional storage and non-hosted engine hosts added etc.
>
> Additional VMs added to hosted-engine storage (oVirt Reports VM and Cinder
> VM). Additional VM's are hosted by other storage - cinder and NFS.
>
> The system is in production.
>
>
> Engine can be migrated around with the web interface.
>
>
> - 3.6.4 upgrade released, follow the upgrade guide, engine is upgraded
> first , new Centos kernel requires host reboot.
>
> - Engine placed on h2 -  h3 into maintenance (local) upgrade and Reboot h3
> - No issues - Local maintenance removed from h3.
>
> - Engine placed on h3 -  h2 into maintenance (local) upgrade and Reboot h2
> - No issues - Local maintenance removed from h2.
>
> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot h1 -
> engine crashes and does not start elsewhere, VM(cinder)  on h3 on same
> gluster volume pauses.
>
> - Host 1 takes about 5 minutes to reboot (Enterprise box with all it's
> normal BIOS probing)
>
> - Engine starts after h1 comes back and stabilises
>
> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole time.
> I can do no diagnosis on the 2 VMs as the engine is not available.
>
> - Local maintenance removed from h​1
>
>
> I don't believe the issue is with gluster itself as the volume remains
> accessible on all hosts during this time albeit with a missing server
> (gluster volume status) as each gluster server is rebooted.
>
> Gluster was upgraded as part of the process, no issues were seen here.
>
>
> I have been able to duplicate the issue without the upgrade by following
> the same sort of timeline.
>
>
> 
> From: Sandro Bonazzola <sbona...@redhat.com>
> Sent: Monday, 11 April 2016 7:11 PM
> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak; Sahina
> Bose
> Cc: Bond, Darryl; users
> Subject: Re: [ovirt-users] Hosted engine on gluster problem
>
>
>
> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck <h...@tbi.univie.ac.at
> <mailto:h...@tbi.univie.ac.at>> wrote:
> Hi Darryl,
>
> I'm still experimenting with my oVirt installation so I tried to
> recreate the problems you've described.
>
> My setup has three HA hosts for virtualization and three machines
> for the gluster replica 3 setup.
>
> I manually migrated the Engine from the initial install host (one)
> to host three. Then shut down host one manually and interrupted the
> fencing mechanisms so the host stayed down. This didn't bother the
> Engine VM at all.
>
> Did you move the host one to maintenance before shutting down?
> Or is this a crash recovery test?
>
>
>
> To make things a bit more challenging I then shut down host three
> while running the Engine VM. Of course the Engine was down for some
> time until host two detected the problem. It started the Engine VM
> and everything seems to be running quite well without the initial
> install host.
>
> Thanks for the feedback!
>
>
>
> My only problem is that the HA agent on host two and three refuse to
> start after a reboot due to the fact that the configuration of the
> hosted engine is missing. I wrote another mail to users@ovirt.org users@ovirt.org>
> about that.
>
> This is weird. Martin,  Simone can you please investigate on this?
>
>
>
>
> Cheers
> Richard
>
> On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> > There seems to be a pretty severe bug with using hosted engine on
> gluster.
> >
> > If the host that was used as the initial hosted-engine --deploy host
> goes away, the engine VM wil crash and cannot be restarted until the host
> comes back.
>
> is this an Hyperconverged setup?
>
>
> >
> >

Re: [ovirt-users] Hosted engine on gluster problem

[Bond, Darryl]

My setup is hyperconverged. I have placed my test results in 
https://bugzilla.redhat.com/show_bug.cgi?id=1298693


Short description of setup:

3 hosts with 2 disks each set up with gluster replica 3 across the 6 disks 
volume name hosted-engine.

Hostname hosted-storage configured in /etc//hosts to point to the host1.

Installed hosted engine on host1 with the hosted engine storage path = 
hosted-storage:/hosted-engine

Install first engine on h1 successful. Hosts h2 and h3 added to the hosted 
engine. All works fine.

Additional storage and non-hosted engine hosts added etc.

Additional VMs added to hosted-engine storage (oVirt Reports VM and Cinder VM). 
Additional VM's are hosted by other storage - cinder and NFS.

The system is in production.


Engine can be migrated around with the web interface.


- 3.6.4 upgrade released, follow the upgrade guide, engine is upgraded first , 
new Centos kernel requires host reboot.

- Engine placed on h2 -  h3 into maintenance (local) upgrade and Reboot h3 - No 
issues - Local maintenance removed from h3.

- Engine placed on h3 -  h2 into maintenance (local) upgrade and Reboot h2 - No 
issues - Local maintenance removed from h2.

- Engine placed on h3 -h1 into mainteance (local) upgrade and reboot h1 - 
engine crashes and does not start elsewhere, VM(cinder)  on h3 on same gluster 
volume pauses.

- Host 1 takes about 5 minutes to reboot (Enterprise box with all it's normal 
BIOS probing)

- Engine starts after h1 comes back and stabilises

- VM(cinder) unpauses itself,  VM(reports) continued fine the whole time. I can 
do no diagnosis on the 2 VMs as the engine is not available.

- Local maintenance removed from h​1


I don't believe the issue is with gluster itself as the volume remains 
accessible on all hosts during this time albeit with a missing server (gluster 
volume status) as each gluster server is rebooted.

Gluster was upgraded as part of the process, no issues were seen here.


I have been able to duplicate the issue without the upgrade by following the 
same sort of timeline.



From: Sandro Bonazzola <sbona...@redhat.com>
Sent: Monday, 11 April 2016 7:11 PM
To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak; Sahina Bose
Cc: Bond, Darryl; users
Subject: Re: [ovirt-users] Hosted engine on gluster problem



On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck 
<h...@tbi.univie.ac.at<mailto:h...@tbi.univie.ac.at>> wrote:
Hi Darryl,

I'm still experimenting with my oVirt installation so I tried to
recreate the problems you've described.

My setup has three HA hosts for virtualization and three machines
for the gluster replica 3 setup.

I manually migrated the Engine from the initial install host (one)
to host three. Then shut down host one manually and interrupted the
fencing mechanisms so the host stayed down. This didn't bother the
Engine VM at all.

Did you move the host one to maintenance before shutting down?
Or is this a crash recovery test?



To make things a bit more challenging I then shut down host three
while running the Engine VM. Of course the Engine was down for some
time until host two detected the problem. It started the Engine VM
and everything seems to be running quite well without the initial
install host.

Thanks for the feedback!



My only problem is that the HA agent on host two and three refuse to
start after a reboot due to the fact that the configuration of the
hosted engine is missing. I wrote another mail to 
users@ovirt.org<mailto:users@ovirt.org>
about that.

This is weird. Martin,  Simone can you please investigate on this?




Cheers
Richard

On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> There seems to be a pretty severe bug with using hosted engine on gluster.
>
> If the host that was used as the initial hosted-engine --deploy host goes 
> away, the engine VM wil crash and cannot be restarted until the host comes 
> back.

is this an Hyperconverged setup?


>
> This is regardless of which host the engine was currently running.
>
>
> The issue seems to be buried in the bowels of VDSM and is not an issue with 
> gluster itself.

Sahina, can you please investigate on this?


>
> The gluster filesystem is still accessable from the host that was running the 
> engine. The issue has been submitted to bugzilla but the fix is some way off 
> (4.1).
>
>
> Can my hosted engine be converted to use NFS (using the gluster NFS server on 
> the same filesystem) without rebuilding my hosted engine (ie change 
> domainType=glusterfs to domainType=nfs)?

>
> What effect would that have on the hosted-engine storage domain inside oVirt, 
> ie would the same filesystem be mounted twice or would it just break.
>
>
> Will this actually fix the problem, does it have the same issue when the 
> hosted engine is on NFS?
>
>
> Darryl
>
>
>
>
> __

Re: [ovirt-users] Hosted engine on gluster problem

[Richard Neuboeck]

On 04/11/2016 11:11 AM, Sandro Bonazzola wrote:
> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
> > wrote:
> 
> Hi Darryl,
> 
> I'm still experimenting with my oVirt installation so I tried to
> recreate the problems you've described.
> 
> My setup has three HA hosts for virtualization and three machines
> for the gluster replica 3 setup.
> 
> I manually migrated the Engine from the initial install host (one)
> to host three. Then shut down host one manually and interrupted the
> fencing mechanisms so the host stayed down. This didn't bother the
> Engine VM at all.
> 
> 
> Did you move the host one to maintenance before shutting down?
> Or is this a crash recovery test?

I did not put the machine in maintenance mode. Just issued
'poweroff' on the command line. Same for host one as host three.

> 
> To make things a bit more challenging I then shut down host three
> while running the Engine VM. Of course the Engine was down for some
> time until host two detected the problem. It started the Engine VM
> and everything seems to be running quite well without the initial
> install host.
> 
> 
> Thanks for the feedback!
> 
>  
> 
> 
> My only problem is that the HA agent on host two and three refuse to
> start after a reboot due to the fact that the configuration of the
> hosted engine is missing. I wrote another mail to
> users@ovirt.org 
> about that.
> 
> 
> This is weird. Martin,  Simone can you please investigate on this?
> 
> 
>  
> 
> 
> Cheers
> Richard
> 
> On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> > There seems to be a pretty severe bug with using hosted engine
> on gluster.
> >
> > If the host that was used as the initial hosted-engine
> --deploy host goes away, the engine VM wil crash and cannot be
> restarted until the host comes back.
> 
> 
> is this an Hyperconverged setup?
> 
>  
> 
> >
> > This is regardless of which host the engine was currently running.
> >
> >
> > The issue seems to be buried in the bowels of VDSM and is not
> an issue with gluster itself.
> 
> 
> Sahina, can you please investigate on this?
> 
>  
> 
> >
> > The gluster filesystem is still accessable from the host that
> was running the engine. The issue has been submitted to bugzilla
> but the fix is some way off (4.1).
> >
> >
> > Can my hosted engine be converted to use NFS (using the
> gluster NFS server on the same filesystem) without rebuilding my
> hosted engine (ie change domainType=glusterfs to domainType=nfs)?
> 
>  
> 
> >
> > What effect would that have on the hosted-engine storage
> domain inside oVirt, ie would the same filesystem be mounted
> twice or would it just break.
> >
> >
> > Will this actually fix the problem, does it have the same
> issue when the hosted engine is on NFS?
> >
> >
> > Darryl
> >
> >
> >
> >
> > 
> >
> > The contents of this electronic message and any attachments
> are intended only for the addressee and may contain legally
> privileged, personal, sensitive or confidential information. If
> you are not the intended addressee, and have received this
> email, any transmission, distribution, downloading, printing or
> photocopying of the contents of this message or attachments is
> strictly prohibited. Any legal privilege or confidentiality
> attached to this message and attachments is not waived, lost or
> destroyed by reason of delivery to any person other than
> intended addressee. If you have received this message and are
> not the intended addressee you should notify the sender by
> return email and destroy all copies of the message and any
> attachments. Unless expressly attributed, the views expressed in
> this email do not necessarily represent the views of the company.
> > ___
> > Users mailing list
> > Users@ovirt.org 
> > http://lists.ovirt.org/mailman/listinfo/users
> >
> 
> 
> --
> /dev/null
> 
> 
> ___
> Users mailing list
> Users@ovirt.org 
> http://lists.ovirt.org/mailman/listinfo/users
> 
> 
> 
> 
> -- 
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community
> collaboration.
> See how it works at redhat.com 


-- 
/dev/null



signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hosted engine on gluster problem

[Sandro Bonazzola]

On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck 
wrote:

> Hi Darryl,
>
> I'm still experimenting with my oVirt installation so I tried to
> recreate the problems you've described.
>
> My setup has three HA hosts for virtualization and three machines
> for the gluster replica 3 setup.
>
> I manually migrated the Engine from the initial install host (one)
> to host three. Then shut down host one manually and interrupted the
> fencing mechanisms so the host stayed down. This didn't bother the
> Engine VM at all.
>

Did you move the host one to maintenance before shutting down?
Or is this a crash recovery test?



>
> To make things a bit more challenging I then shut down host three
> while running the Engine VM. Of course the Engine was down for some
> time until host two detected the problem. It started the Engine VM
> and everything seems to be running quite well without the initial
> install host.
>

Thanks for the feedback!



>
> My only problem is that the HA agent on host two and three refuse to
> start after a reboot due to the fact that the configuration of the
> hosted engine is missing. I wrote another mail to users@ovirt.org
> about that.
>

This is weird. Martin,  Simone can you please investigate on this?




>
> Cheers
> Richard
>
> On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> > There seems to be a pretty severe bug with using hosted engine on
> gluster.
> >
> > If the host that was used as the initial hosted-engine --deploy host
> goes away, the engine VM wil crash and cannot be restarted until the host
> comes back.
>

is this an Hyperconverged setup?



> >
> > This is regardless of which host the engine was currently running.
> >
> >
> > The issue seems to be buried in the bowels of VDSM and is not an issue
> with gluster itself.
>

Sahina, can you please investigate on this?



> >
> > The gluster filesystem is still accessable from the host that was
> running the engine. The issue has been submitted to bugzilla but the fix is
> some way off (4.1).
> >
> >
> > Can my hosted engine be converted to use NFS (using the gluster NFS
> server on the same filesystem) without rebuilding my hosted engine (ie
> change domainType=glusterfs to domainType=nfs)?
>

>
>
> > What effect would that have on the hosted-engine storage domain inside
> oVirt, ie would the same filesystem be mounted twice or would it just break.
> >
> >
> > Will this actually fix the problem, does it have the same issue when the
> hosted engine is on NFS?
> >
> >
> > Darryl
> >
> >
> >
> >
> > 
> >
> > The contents of this electronic message and any attachments are intended
> only for the addressee and may contain legally privileged, personal,
> sensitive or confidential information. If you are not the intended
> addressee, and have received this email, any transmission, distribution,
> downloading, printing or photocopying of the contents of this message or
> attachments is strictly prohibited. Any legal privilege or confidentiality
> attached to this message and attachments is not waived, lost or destroyed
> by reason of delivery to any person other than intended addressee. If you
> have received this message and are not the intended addressee you should
> notify the sender by return email and destroy all copies of the message and
> any attachments. Unless expressly attributed, the views expressed in this
> email do not necessarily represent the views of the company.
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
>
> --
> /dev/null
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hosted engine on gluster problem

[Richard Neuboeck]

Hi Darryl,

I'm still experimenting with my oVirt installation so I tried to
recreate the problems you've described.

My setup has three HA hosts for virtualization and three machines
for the gluster replica 3 setup.

I manually migrated the Engine from the initial install host (one)
to host three. Then shut down host one manually and interrupted the
fencing mechanisms so the host stayed down. This didn't bother the
Engine VM at all.

To make things a bit more challenging I then shut down host three
while running the Engine VM. Of course the Engine was down for some
time until host two detected the problem. It started the Engine VM
and everything seems to be running quite well without the initial
install host.

My only problem is that the HA agent on host two and three refuse to
start after a reboot due to the fact that the configuration of the
hosted engine is missing. I wrote another mail to users@ovirt.org
about that.

Cheers
Richard

On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> There seems to be a pretty severe bug with using hosted engine on gluster.
> 
> If the host that was used as the initial hosted-engine --deploy host goes 
> away, the engine VM wil crash and cannot be restarted until the host comes 
> back.
> 
> This is regardless of which host the engine was currently running.
> 
> 
> The issue seems to be buried in the bowels of VDSM and is not an issue with 
> gluster itself.
> 
> The gluster filesystem is still accessable from the host that was running the 
> engine. The issue has been submitted to bugzilla but the fix is some way off 
> (4.1).
> 
> 
> Can my hosted engine be converted to use NFS (using the gluster NFS server on 
> the same filesystem) without rebuilding my hosted engine (ie change 
> domainType=glusterfs to domainType=nfs)?
> 
> What effect would that have on the hosted-engine storage domain inside oVirt, 
> ie would the same filesystem be mounted twice or would it just break.
> 
> 
> Will this actually fix the problem, does it have the same issue when the 
> hosted engine is on NFS?
> 
> 
> Darryl
> 
> 
> 
> 
> 
> 
> The contents of this electronic message and any attachments are intended only 
> for the addressee and may contain legally privileged, personal, sensitive or 
> confidential information. If you are not the intended addressee, and have 
> received this email, any transmission, distribution, downloading, printing or 
> photocopying of the contents of this message or attachments is strictly 
> prohibited. Any legal privilege or confidentiality attached to this message 
> and attachments is not waived, lost or destroyed by reason of delivery to any 
> person other than intended addressee. If you have received this message and 
> are not the intended addressee you should notify the sender by return email 
> and destroy all copies of the message and any attachments. Unless expressly 
> attributed, the views expressed in this email do not necessarily represent 
> the views of the company.
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 


-- 
/dev/null



signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users