Re: [ovirt-users] global vs local maintenance with single host

[Gianluca Cecchi]

On Sun, Sep 4, 2016 at 10:54 AM, Yedidyah Bar David  wrote:

> On Sat, Sep 3, 2016 at 1:18 PM, Gianluca Cecchi
>  wrote:
> > Hello,
> > how do the two modes apply in case of single host?
> > During an upgrade phase, after having upgraded the self hosted engine and
> > leaving global maintenance and having checked all is ok, what is the
> correct
> > mode then to put host if I want finally to update it too?
>
> The docs say to put hosts to maintenance from the engine before upgrading
> them.
>
> This is (also) so that VMs on them are migrated away to other hosts.
>
> With a single host, you have no other hosts to migrate VMs to.
>
> So you should do something like this:
>
> 1. Set global maintenance (because you are going to take down the
> engine and its vm)
> 2. Shutdown all other VMs
> 3. Shutdown engine vm from itself
> At this point, you should be able to simply stop HA services. But it
> might be cleaner to first set local maintenance. Not sure but perhaps
> this might be required for vdsm. So:
> 4. Set local maintenance
> 5. Stop HA services. If setting local maintenance didn't work, perhaps
> better stop also vdsm services. This stop should obviously happen
> automatically by yum/rpm, but perhaps better do this manually to see
> that it worked.
> 6. yum (or dnf) update stuff.
> 7. Start HA services
> 8. Check status. I think you'll see that both local and global maint
> are still set.
> 9. Set maintenance to none
> 10. Check status again - I think that after some time HA will decide
> to start engine vm and should succeed.
> 11. Start all other VMs.
>
> Didn't try this myself.
>
> Best,
> --
> Didi
>

I tested on one of the 2 environments.
It seems it worked.
But I update the kernel on host without restarting it. I would try that
with the other one.
Some notes:

8. Check status. I think you'll see that both local and global maint
are still set.

Actually even if I'm on global maintenance and then I set local
maintenance, it seems I "loose" the global maintenance state...

I see this output, without the line with Global Maintenance and exclamation
marks:

[root@ractor ~]# hosted-engine --vm-status
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py:15:
DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is
deprecated, please use vdsm.jsonrpcvdscli
  import vdsm.vdscli


--== Host 1 status ==--

Status up-to-date  : False
Hostname   : ractor.mydomain
Host ID: 1
Engine status  : unknown stale-data
Score  : 0
stopped: False
Local maintenance  : True
crc32  : d616dde1
Host timestamp : 3304360
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=3304360 (Mon Oct  3 22:27:07 2016)
host-id=1
score=0
maintenance=True
state=LocalMaintenance
stopped=False
[root@ractor ~]#


I'm able to exit maintenance, connect to engine and start the other VMs.
Now I have to try considering also the restart of the hypervisor host, due
to new kernel package install.

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] global vs local maintenance with single host

[Gervais de Montbrun]

Hi Gianluca,

I forgot to mention that you need to ensure that systemd knows that the new 
file exists. You should likely run `systemctl daemon-reload` after 
creating/modifying your custom systemd files. You can see that the After 
directive is combined from both files. Check it out by running `systemctl show 
vdsmd.service | grep After`

It makes sense to make further changes to ensure that NFS stops last, but I 
haven't looked into that yet.
:-)

Cheers,
Gervais



> On Oct 3, 2016, at 7:22 AM, Gianluca Cecchi  wrote:
> 
> 
> Il 28/Set/2016 21:09, "Gervais de Montbrun"  > ha scritto:
> >
> > Hi Gianluca,
> >
> > Instead of editing the system's built in systemd configuration, you can do 
> > the following...
> >
> > Create a file called /etc/systemd/system/ovirt-ha-broker.service
> >
> >> # My custom ovirt-ha-broker.service config that ensures NFS starts before 
> >> ovirt-ha-broker.service
> >> # thanks Gervais for this tip!  :-)
> >>
> >> .include /usr/lib/systemd/system/ovirt-ha-broker.service
> >>
> >> [Unit]
> >> After=nfs-server.service
> >
> >
> > Then disable and enable ovirt-ha-broker.service (systemctl disable 
> > ovirt-ha-broker.service ; systemctl enable ovirt-ha-broker.service) and you 
> > should see that it is using your customized systemd unit definition. You 
> > can see that systemd is using your file by running systemctl status 
> > ovirt-ha-broker.service. You'll see something like "Loaded: loaded 
> > (/etc/systemd/system/ovirt-ha-broker.service;" in the output.
> >
> > Your file will survive updates and therefore always wait for nfs to start 
> > prior to starting. You can do the same for your other customizations.
> >
> > Cheers,
> > Gervais
> >
> >
> >
> >> On Sep 28, 2016, at 1:31 PM, Gianluca Cecchi  >> > wrote:
> >>
> >> On Sun, Sep 4, 2016 at 10:54 AM, Yedidyah Bar David  >> > wrote:
> >>>
> >>> On Sat, Sep 3, 2016 at 1:18 PM, Gianluca Cecchi
> >>> > wrote:
> >>> > Hello,
> >>> > how do the two modes apply in case of single host?
> >>> > During an upgrade phase, after having upgraded the self hosted engine 
> >>> > and
> >>> > leaving global maintenance and having checked all is ok, what is the 
> >>> > correct
> >>> > mode then to put host if I want finally to update it too?
> >>>
> >>> The docs say to put hosts to maintenance from the engine before upgrading 
> >>> them.
> >>>
> >>> This is (also) so that VMs on them are migrated away to other hosts.
> >>>
> >>> With a single host, you have no other hosts to migrate VMs to.
> >>>
> >>> So you should do something like this:
> >>>
> >>> 1. Set global maintenance (because you are going to take down the
> >>> engine and its vm)
> >>> 2. Shutdown all other VMs
> >>> 3. Shutdown engine vm from itself
> >>> At this point, you should be able to simply stop HA services. But it
> >>> might be cleaner to first set local maintenance. Not sure but perhaps
> >>> this might be required for vdsm. So:
> >>> 4. Set local maintenance
> >>> 5. Stop HA services. If setting local maintenance didn't work, perhaps
> >>> better stop also vdsm services. This stop should obviously happen
> >>> automatically by yum/rpm, but perhaps better do this manually to see
> >>> that it worked.
> >>> 6. yum (or dnf) update stuff.
> >>> 7. Start HA services
> >>> 8. Check status. I think you'll see that both local and global maint
> >>> are still set.
> >>> 9. Set maintenance to none
> >>> 10. Check status again - I think that after some time HA will decide
> >>> to start engine vm and should succeed.
> >>> 11. Start all other VMs.
> >>>
> >>> Didn't try this myself.
> >>>
> >>> Best,
> >>> --
> >>> Didi
> >>
> >>
> >> Hello Didi,
> >> I would like to leverage the update I have to do on 2 small different lab 
> >> environments to crosscheck the steps suggested.
> >> They are both single host environments with self hosted engine.
> >> One is 4.0.2 and the other is 4.0.3. Both on CentoS 7.2
> >> I plan to migrate to the just released 4.0.4
> >>
> >> One note: in both environments the storage is NFS and is provided by the 
> >> host itself, so a corner case (for all hosted_storage domain, main data 
> >> domain and iso storage domain).
> >> I customized the init scripts, basically for start phase of the server and 
> >> to keep in count of the NFS service, but probably something has to be done 
> >> for stop too?
> >>
> >> 1) In /usr/lib/systemd/system/ovirt-ha-broker.service
> >>
> >> added in section [Unit]
> >>
> >> After=nfs-server.service
> >>
> >> The file is overwritten at update so one has to keep in mind this
> >>
> >> 2) also in vdsmd.service changed 
> >> from:
> >> After=multipathd.service libvirtd.service iscsid.service rpcbind.service \
> >>   supervdsmd.service sanlock.service vdsm-network.service
> 

Re: [ovirt-users] global vs local maintenance with single host

[Gianluca Cecchi]

Il 28/Set/2016 21:09, "Gervais de Montbrun"  ha
scritto:
>
> Hi Gianluca,
>
> Instead of editing the system's built in systemd configuration, you can
do the following...
>
> Create a file called /etc/systemd/system/ovirt-ha-broker.service
>
>> # My custom ovirt-ha-broker.service config that ensures NFS starts
before ovirt-ha-broker.service
>> # thanks Gervais for this tip!  :-)
>>
>> .include /usr/lib/systemd/system/ovirt-ha-broker.service
>>
>> [Unit]
>> After=nfs-server.service
>
>
> Then disable and enable ovirt-ha-broker.service (systemctl disable
ovirt-ha-broker.service ; systemctl enable ovirt-ha-broker.service) and you
should see that it is using your customized systemd unit definition. You
can see that systemd is using your file by running systemctl status
ovirt-ha-broker.service. You'll see something like "Loaded: loaded
(/etc/systemd/system/ovirt-ha-broker.service;" in the output.
>
> Your file will survive updates and therefore always wait for nfs to start
prior to starting. You can do the same for your other customizations.
>
> Cheers,
> Gervais
>
>
>
>> On Sep 28, 2016, at 1:31 PM, Gianluca Cecchi 
wrote:
>>
>> On Sun, Sep 4, 2016 at 10:54 AM, Yedidyah Bar David 
wrote:
>>>
>>> On Sat, Sep 3, 2016 at 1:18 PM, Gianluca Cecchi
>>>  wrote:
>>> > Hello,
>>> > how do the two modes apply in case of single host?
>>> > During an upgrade phase, after having upgraded the self hosted engine
and
>>> > leaving global maintenance and having checked all is ok, what is the
correct
>>> > mode then to put host if I want finally to update it too?
>>>
>>> The docs say to put hosts to maintenance from the engine before
upgrading them.
>>>
>>> This is (also) so that VMs on them are migrated away to other hosts.
>>>
>>> With a single host, you have no other hosts to migrate VMs to.
>>>
>>> So you should do something like this:
>>>
>>> 1. Set global maintenance (because you are going to take down the
>>> engine and its vm)
>>> 2. Shutdown all other VMs
>>> 3. Shutdown engine vm from itself
>>> At this point, you should be able to simply stop HA services. But it
>>> might be cleaner to first set local maintenance. Not sure but perhaps
>>> this might be required for vdsm. So:
>>> 4. Set local maintenance
>>> 5. Stop HA services. If setting local maintenance didn't work, perhaps
>>> better stop also vdsm services. This stop should obviously happen
>>> automatically by yum/rpm, but perhaps better do this manually to see
>>> that it worked.
>>> 6. yum (or dnf) update stuff.
>>> 7. Start HA services
>>> 8. Check status. I think you'll see that both local and global maint
>>> are still set.
>>> 9. Set maintenance to none
>>> 10. Check status again - I think that after some time HA will decide
>>> to start engine vm and should succeed.
>>> 11. Start all other VMs.
>>>
>>> Didn't try this myself.
>>>
>>> Best,
>>> --
>>> Didi
>>
>>
>> Hello Didi,
>> I would like to leverage the update I have to do on 2 small different
lab environments to crosscheck the steps suggested.
>> They are both single host environments with self hosted engine.
>> One is 4.0.2 and the other is 4.0.3. Both on CentoS 7.2
>> I plan to migrate to the just released 4.0.4
>>
>> One note: in both environments the storage is NFS and is provided by the
host itself, so a corner case (for all hosted_storage domain, main data
domain and iso storage domain).
>> I customized the init scripts, basically for start phase of the server
and to keep in count of the NFS service, but probably something has to be
done for stop too?
>>
>> 1) In /usr/lib/systemd/system/ovirt-ha-broker.service
>>
>> added in section [Unit]
>>
>> After=nfs-server.service
>>
>> The file is overwritten at update so one has to keep in mind this
>>
>> 2) also in vdsmd.service changed
>> from:
>> After=multipathd.service libvirtd.service iscsid.service rpcbind.service
\
>>   supervdsmd.service sanlock.service vdsm-network.service
>>
>> to:
>> After=multipathd.service libvirtd.service iscsid.service rpcbind.service
\
>>   supervdsmd.service sanlock.service vdsm-network.service \
>>   nfs-server.service
>>
>> Do you think any order setup I have to put in place related to NFS
service and oVirt services stop?
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>

Nice! I'm going to try and see.
Any particular dependency I should add for shutdown order due to the fact
that my host is also the NFS server providing data stores?
Do I need to set up nfs stop only after a particular ovirt related service?
Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] global vs local maintenance with single host

[Gervais de Montbrun]

Hi Gianluca,

Instead of editing the system's built in systemd configuration, you can do the 
following...

Create a file called /etc/systemd/system/ovirt-ha-broker.service

# My custom ovirt-ha-broker.service config that ensures NFS starts before 
ovirt-ha-broker.service
# thanks Gervais for this tip!  :-)

.include /usr/lib/systemd/system/ovirt-ha-broker.service

[Unit]
After=nfs-server.service

Then disable and enable ovirt-ha-broker.service (systemctl disable 
ovirt-ha-broker.service ; systemctl enable ovirt-ha-broker.service) and you 
should see that it is using your customized systemd unit definition. You can 
see that systemd is using your file by running systemctl status 
ovirt-ha-broker.service. You'll see something like "Loaded: loaded 
(/etc/systemd/system/ovirt-ha-broker.service;" in the output.

Your file will survive updates and therefore always wait for nfs to start prior 
to starting. You can do the same for your other customizations.

Cheers,
Gervais



> On Sep 28, 2016, at 1:31 PM, Gianluca Cecchi  
> wrote:
> 
> On Sun, Sep 4, 2016 at 10:54 AM, Yedidyah Bar David  > wrote:
> On Sat, Sep 3, 2016 at 1:18 PM, Gianluca Cecchi
> > wrote:
> > Hello,
> > how do the two modes apply in case of single host?
> > During an upgrade phase, after having upgraded the self hosted engine and
> > leaving global maintenance and having checked all is ok, what is the correct
> > mode then to put host if I want finally to update it too?
> 
> The docs say to put hosts to maintenance from the engine before upgrading 
> them.
> 
> This is (also) so that VMs on them are migrated away to other hosts.
> 
> With a single host, you have no other hosts to migrate VMs to.
> 
> So you should do something like this:
> 
> 1. Set global maintenance (because you are going to take down the
> engine and its vm)
> 2. Shutdown all other VMs
> 3. Shutdown engine vm from itself
> At this point, you should be able to simply stop HA services. But it
> might be cleaner to first set local maintenance. Not sure but perhaps
> this might be required for vdsm. So:
> 4. Set local maintenance
> 5. Stop HA services. If setting local maintenance didn't work, perhaps
> better stop also vdsm services. This stop should obviously happen
> automatically by yum/rpm, but perhaps better do this manually to see
> that it worked.
> 6. yum (or dnf) update stuff.
> 7. Start HA services
> 8. Check status. I think you'll see that both local and global maint
> are still set.
> 9. Set maintenance to none
> 10. Check status again - I think that after some time HA will decide
> to start engine vm and should succeed.
> 11. Start all other VMs.
> 
> Didn't try this myself.
> 
> Best,
> --
> Didi
> 
> Hello Didi,
> I would like to leverage the update I have to do on 2 small different lab 
> environments to crosscheck the steps suggested.
> They are both single host environments with self hosted engine.
> One is 4.0.2 and the other is 4.0.3. Both on CentoS 7.2
> I plan to migrate to the just released 4.0.4
> 
> One note: in both environments the storage is NFS and is provided by the host 
> itself, so a corner case (for all hosted_storage domain, main data domain and 
> iso storage domain).
> I customized the init scripts, basically for start phase of the server and to 
> keep in count of the NFS service, but probably something has to be done for 
> stop too?
> 
> 1) In /usr/lib/systemd/system/ovirt-ha-broker.service
> 
> added in section [Unit]
> 
> After=nfs-server.service
> 
> The file is overwritten at update so one has to keep in mind this
> 
> 2) also in vdsmd.service changed 
> from:
> After=multipathd.service libvirtd.service iscsid.service rpcbind.service \
>   supervdsmd.service sanlock.service vdsm-network.service
> 
> to:
> After=multipathd.service libvirtd.service iscsid.service rpcbind.service \
>   supervdsmd.service sanlock.service vdsm-network.service \
>   nfs-server.service
> 
> Do you think any order setup I have to put in place related to NFS service 
> and oVirt services stop?
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] global vs local maintenance with single host

[Gianluca Cecchi]

On Sun, Sep 4, 2016 at 10:54 AM, Yedidyah Bar David  wrote:

> On Sat, Sep 3, 2016 at 1:18 PM, Gianluca Cecchi
>  wrote:
> > Hello,
> > how do the two modes apply in case of single host?
> > During an upgrade phase, after having upgraded the self hosted engine and
> > leaving global maintenance and having checked all is ok, what is the
> correct
> > mode then to put host if I want finally to update it too?
>
> The docs say to put hosts to maintenance from the engine before upgrading
> them.
>
> This is (also) so that VMs on them are migrated away to other hosts.
>
> With a single host, you have no other hosts to migrate VMs to.
>
> So you should do something like this:
>
> 1. Set global maintenance (because you are going to take down the
> engine and its vm)
> 2. Shutdown all other VMs
> 3. Shutdown engine vm from itself
> At this point, you should be able to simply stop HA services. But it
> might be cleaner to first set local maintenance. Not sure but perhaps
> this might be required for vdsm. So:
> 4. Set local maintenance
> 5. Stop HA services. If setting local maintenance didn't work, perhaps
> better stop also vdsm services. This stop should obviously happen
> automatically by yum/rpm, but perhaps better do this manually to see
> that it worked.
> 6. yum (or dnf) update stuff.
> 7. Start HA services
> 8. Check status. I think you'll see that both local and global maint
> are still set.
> 9. Set maintenance to none
> 10. Check status again - I think that after some time HA will decide
> to start engine vm and should succeed.
> 11. Start all other VMs.
>
> Didn't try this myself.
>
> Best,
> --
> Didi
>

Hello Didi,
I would like to leverage the update I have to do on 2 small different lab
environments to crosscheck the steps suggested.
They are both single host environments with self hosted engine.
One is 4.0.2 and the other is 4.0.3. Both on CentoS 7.2
I plan to migrate to the just released 4.0.4

One note: in both environments the storage is NFS and is provided by the
host itself, so a corner case (for all hosted_storage domain, main data
domain and iso storage domain).
I customized the init scripts, basically for start phase of the server and
to keep in count of the NFS service, but probably something has to be done
for stop too?

1) In /usr/lib/systemd/system/ovirt-ha-broker.service

added in section [Unit]

After=nfs-server.service

The file is overwritten at update so one has to keep in mind this

2) also in vdsmd.service changed
from:
After=multipathd.service libvirtd.service iscsid.service rpcbind.service \
  supervdsmd.service sanlock.service vdsm-network.service

to:
After=multipathd.service libvirtd.service iscsid.service rpcbind.service \
  supervdsmd.service sanlock.service vdsm-network.service \
  nfs-server.service

Do you think any order setup I have to put in place related to NFS service
and oVirt services stop?
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] global vs local maintenance with single host

[Yedidyah Bar David]

On Sat, Sep 3, 2016 at 1:18 PM, Gianluca Cecchi
 wrote:
> Hello,
> how do the two modes apply in case of single host?
> During an upgrade phase, after having upgraded the self hosted engine and
> leaving global maintenance and having checked all is ok, what is the correct
> mode then to put host if I want finally to update it too?

The docs say to put hosts to maintenance from the engine before upgrading them.

This is (also) so that VMs on them are migrated away to other hosts.

With a single host, you have no other hosts to migrate VMs to.

So you should do something like this:

1. Set global maintenance (because you are going to take down the
engine and its vm)
2. Shutdown all other VMs
3. Shutdown engine vm from itself
At this point, you should be able to simply stop HA services. But it
might be cleaner to first set local maintenance. Not sure but perhaps
this might be required for vdsm. So:
4. Set local maintenance
5. Stop HA services. If setting local maintenance didn't work, perhaps
better stop also vdsm services. This stop should obviously happen
automatically by yum/rpm, but perhaps better do this manually to see
that it worked.
6. yum (or dnf) update stuff.
7. Start HA services
8. Check status. I think you'll see that both local and global maint
are still set.
9. Set maintenance to none
10. Check status again - I think that after some time HA will decide
to start engine vm and should succeed.
11. Start all other VMs.

Didn't try this myself.

Best,
-- 
Didi
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] global vs local maintenance with single host

[Gianluca Cecchi]

Hello,
how do the two modes apply in case of single host?
During an upgrade phase, after having upgraded the self hosted engine and
leaving global maintenance and having checked all is ok, what is the
correct mode then to put host if I want finally to update it too?

Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users