Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Chris]

Hello Tzach,

Agree for a graceful shutdown there should be some steps ahead. Im 
actually asking the question because the hardware we use has a high 
failure rate on RAID controllers. The physical machine reacts with a 
reboot, so its not controlable from our side. We dont want to delete the 
instances at all, its just about the Instance shutdown/process kill.


Got a good answer already:
If you are using CentOS  / Red Hat you can modify this file:
/etc/sysconfig/libvirt-guests
libvirt-guests:#ON_SHUTDOWN=suspend


Thanks for your answer!

Cheers Chris


On 2017-11-02 15:54, Tzach Shefi wrote:

Hi, 

A better Q would be why do you shutdown a compute node to begin with?
I mean if you need you should do so in an orderly fashion basically
excavate instances

or shut instances down manually, put the compute node in maintenance
mode. 
On rebooting compute node remove it from maintenance mode, turn on
instances or migrate them back to this compute node should you need. 
Od delete them if you wish. 

There is this nova option:
resume_guests_state_on_host_boot=true

But it doesn't delete or shutdown instances but rather turns them on
automatically once compute host resumes. 
which might also work for you, probably not just mentioning it any
way.  

I don't know of an option to stop/delete instance on compute node
shutdown. 

Another option check maybe you could shelve suspended instance and
then later delete them. 

Shelving stops the instance and takes a snapshot of it. Then depending
on the value of the shelved_offload_time config option, the instance
is deleted from the hypervisor (0), never deleted (-1), or deleted
after some period of time (> 0). Note that it's just destroying the
backing instance on the hypervisor, the actual instance in the nova
database is not deleted. Then you can later unshelve the instance:

This might help, but do not if you mess with kvm without updating Nova
you might be left haning else where :)
https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/
[2]

 

On Thu, Nov 2, 2017 at 9:03 AM, Chris  wrote:


Hello,

When we shut down a compute node the instances running on it get
suspended. This generates some difficulties with some applications
like RabbitMQ dont like to be suspended. Is there a way to change
this behavior so that the running instances gets killed or shutdown
instead?

Thanks in advance.

Cheers,
Chris

___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]


--

Tzach Shefi

Senior Quality Engineer, RHCSA

Red Hat

 tsh...@redaht.com    M: +972-54-4701080 [3]     IM: tshefi

 [4]



Links:
--
[1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[2]
https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/
[3] tel:+972-54-4701080
[4] https://red.ht/sig


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Chris]

Hello Remo,

Exactly what I was looking for! If I could give you a rating it would be 
the highest :)


Cheers,
Chris

On 2017-11-03 1:27, Remo Mattei wrote:

sorry forgot to paste the file location
/etc/sysconfig/libvirt-guests




On 11/2/17 11:21 AM, Remo Mattei wrote:

If you are using CentOS  / Red Hat you can modify this file:

libvirt-guests:#ON_SHUTDOWN=suspend



# action taken on host shutdown
# - suspend   all running guests are suspended using virsh managedsave
# - shutdown  all running guests are asked to shutdown. Please be
careful with
# this settings since there is no way to distinguish 
between a
# guest which is stuck or ignores shutdown requests and a 
guest

# which just needs a long time to shutdown. When setting
# ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT 
to a

# value suitable for your guests.
#ON_SHUTDOWN=suspend



Hope this help.

Remo

On 11/2/17 11:08 AM, Chris Friesen wrote:

On 11/02/2017 01:03 AM, Chris wrote:

Hello,

When we shut down a compute node the instances running on it get
suspended. This
generates some difficulties with some applications like RabbitMQ 
dont

like to be
suspended. Is there a way to change this behavior so that the 
running

instances
gets killed or shutdown instead?
This may be done by the libvirtd shutdown scripts rather than 
anything

in nova.

As others have said, you should probably either shut down the VMs in
an orderly fashion or else cold/live migrate the instances off the
compute node before rebooting it.

Chris

___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Post to : openstack@lists.openstack.org
Unsubscribe : 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



___
Mailing list: 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Post to : openstack@lists.openstack.org
Unsubscribe : 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Van Leeuwen, Robert]

>There is this nova option:
>resume_guests_state_on_host_boot=true
>
>But it doesn't delete or shutdown instances but rather turns them on 
>automatically once compute host resumes.
>which might also work for you, probably not just mentioning it any way.

Note that you might want to check some things after a reboot before booting up 
all your instances.
The easiest way would be to set this value to true but to not start the 
nova-compute service on boot.
Now you can check whatever you need and start nova-compute when you know the 
server is in a good state.
Nova will now start the instances that where running when your compute node was 
shutdown.
The nice thing about this is that any instances that where shutdown/suspended 
by users will keep in a shutdown state.
If you do not do this you would need to create a custom script that saves which 
instances where running before the shutdown and then use that list to start 
only the correct instances.

Another thing to consider is if you want to auto-start instances after a 
compute node did an unexpected reboot.
IMHO I would rather have a dead instance then an instance running on flaky 
hard/software but it might depend a bit on your use case.

>I don't know of an option to stop/delete instance on compute node shutdown.
As mentioned by some other people this is actually a libvirt setting not a nova 
setting.


Cheers,
Robert van Leeuwen
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Remo Mattei]

sorry forgot to paste the file location
/etc/sysconfig/libvirt-guests




On 11/2/17 11:21 AM, Remo Mattei wrote:
> If you are using CentOS  / Red Hat you can modify this file:
>
> libvirt-guests:#ON_SHUTDOWN=suspend
>
>
>
> # action taken on host shutdown
> # - suspend   all running guests are suspended using virsh managedsave
> # - shutdown  all running guests are asked to shutdown. Please be
> careful with
> # this settings since there is no way to distinguish between a
> # guest which is stuck or ignores shutdown requests and a guest
> # which just needs a long time to shutdown. When setting
> # ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a
> # value suitable for your guests.
> #ON_SHUTDOWN=suspend
>
>
>
> Hope this help.
>
> Remo
>
> On 11/2/17 11:08 AM, Chris Friesen wrote:
>> On 11/02/2017 01:03 AM, Chris wrote:
>>> Hello,
>>>
>>> When we shut down a compute node the instances running on it get
>>> suspended. This
>>> generates some difficulties with some applications like RabbitMQ dont
>>> like to be
>>> suspended. Is there a way to change this behavior so that the running
>>> instances
>>> gets killed or shutdown instead?
>> This may be done by the libvirtd shutdown scripts rather than anything
>> in nova.
>>
>> As others have said, you should probably either shut down the VMs in
>> an orderly fashion or else cold/live migrate the instances off the
>> compute node before rebooting it.
>>
>> Chris
>>
>> ___
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Remo Mattei]

If you are using CentOS  / Red Hat you can modify this file:

libvirt-guests:#ON_SHUTDOWN=suspend



# action taken on host shutdown
# - suspend   all running guests are suspended using virsh managedsave
# - shutdown  all running guests are asked to shutdown. Please be
careful with
# this settings since there is no way to distinguish between a
# guest which is stuck or ignores shutdown requests and a guest
# which just needs a long time to shutdown. When setting
# ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a
# value suitable for your guests.
#ON_SHUTDOWN=suspend



Hope this help.

Remo

On 11/2/17 11:08 AM, Chris Friesen wrote:
> On 11/02/2017 01:03 AM, Chris wrote:
>> Hello,
>>
>> When we shut down a compute node the instances running on it get
>> suspended. This
>> generates some difficulties with some applications like RabbitMQ dont
>> like to be
>> suspended. Is there a way to change this behavior so that the running
>> instances
>> gets killed or shutdown instead?
>
> This may be done by the libvirtd shutdown scripts rather than anything
> in nova.
>
> As others have said, you should probably either shut down the VMs in
> an orderly fashion or else cold/live migrate the instances off the
> compute node before rebooting it.
>
> Chris
>
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Chris Friesen]

On 11/02/2017 01:03 AM, Chris wrote:

Hello,

When we shut down a compute node the instances running on it get suspended. This
generates some difficulties with some applications like RabbitMQ dont like to be
suspended. Is there a way to change this behavior so that the running instances
gets killed or shutdown instead?


This may be done by the libvirtd shutdown scripts rather than anything in nova.

As others have said, you should probably either shut down the VMs in an orderly 
fashion or else cold/live migrate the instances off the compute node before 
rebooting it.


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Tomáš Vondra]

Hi!

When I need to reboot a compute node (because of some driver lock-up or such 
problems), I first stop nova-compute so that it does not report the shutoff 
state of VMs to the database and resume_guests_state_on_host_boot=true does 
actually start them. Then I press the power switch on the VMs using virsh 
shutdown, one by one, and only after that I reboot the node. This is a script I 
scrounged somewhere and modified so that it does not take too much time:

Tomas

 

root@cmp03:~# cat vm-shutdown

#!/bin/bash

# file: /usr/local/sbin/vm-shutdown

# Description: shutdown active virtual machines

debug=1

#fake=1

 

# Get list of active virtual machines

vmList="`virsh list | (

while read vmID vmName vmStatus

do

  if [ -n "$vmName" -a "$vmName" != "Name" -a "$vmName" != "Domain-0" ]

  then

[ -z "$vmList" ] && vmList="$vmName" || vmList="$vmList $vmName"

  fi

done

echo $vmList )`"

 

# check there are some active VM's

if [ -n "$vmList" ]; then

  # Shutdown VM's with verification

  for vmName in $vmList

  do

# send initial request

[ -n "$debug" ] && echo -n "Attempting to shutdown $vmName "

[ -z "$fake" ] && virsh shutdown $vmName

# wait a limited time for the VM to be not running

count=30

while $( virsh list | grep $vmName >/dev/null ) && [ $count -gt 0 ]

do

  sleep 1

  let count=count-1

  [ -n "$debug" ] && echo -n "."

done

# report current status

( virsh list | grep $vmName >/dev/null ) && echo " failed!" || echo "down."

# if still running, destroy it

if ( virsh list | grep $vmName >/dev/null )

then

  [ -n "$debug" ] && echo -n "Attempting to destroy $vmName "

  [ -z "$fake" ] && virsh destroy $vmName

  # wait a limited time for the VM to be not running

  count=30

  while $( virsh list | grep $vmName >/dev/null ) && [ $count -gt 0 ]

  do

sleep 1

let count=count-1

[ -n "$debug" ] && echo -n "."

      done

      # report current status

  ( virsh list | grep $vmName >/dev/null ) && echo " failed!" || echo 
"down."

fi

  done

 

From: Tzach Shefi [mailto:tsh...@redhat.com] 
Sent: Thursday, November 02, 2017 9:55 AM
To: Chris
Cc: openstack@lists.openstack.org
Subject: Re: [Openstack] Compute Node shutdown how to prevent instance suspend

 

Hi, 

 

A better Q would be why do you shutdown a compute node to begin with?

I mean if you need you should do so in an orderly fashion basically excavate 
instances

or shut instances down manually, put the compute node in maintenance mode. 

On rebooting compute node remove it from maintenance mode, turn on instances or 
migrate them back to this compute node should you need. 

Od delete them if you wish. 

 

There is this nova option:

resume_guests_state_on_host_boot=true

 

But it doesn't delete or shutdown instances but rather turns them on 
automatically once compute host resumes. 

which might also work for you, probably not just mentioning it any way.  

 

I don't know of an option to stop/delete instance on compute node shutdown. 

 

Another option check maybe you could shelve suspended instance and then later 
delete them. 

 

Shelving stops the instance and takes a snapshot of it. Then depending on the 
value of the shelved_offload_time config option, the instance is deleted from 
the hypervisor (0), never deleted (-1), or deleted after some period of time (> 
0). Note that it's just destroying the backing instance on the hypervisor, the 
actual instance in the nova database is not deleted. Then you can later 
unshelve the instance:

 

 

This might help, but do not if you mess with kvm without updating Nova you 
might be left haning else where :)

https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/

 

 

 

 

On Thu, Nov 2, 2017 at 9:03 AM, Chris <cont...@progbau.de> wrote:

Hello,

When we shut down a compute node the instances running on it get suspended. 
This generates some difficulties with some applications like RabbitMQ dont like 
to be suspended. Is there a way to change this behavior so that the running 
instances gets killed or shutdown instead?

Thanks in advance.

Cheers,
Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack





 

-- 

Tzach Shefi

Senior Quality Engineer, RHCSA

 <https://www.redhat.com> Red Hat 



 <mailto:tsh...@redaht.com> tsh...@redaht.comM:  <tel:+972-54-4701080> 
+972-54-4701080 IM: tshefi


 <https://red.ht/sig> Obrázek byl odebrán odesílatelem.

 

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Tzach Shefi]

Hi,

A better Q would be why do you shutdown a compute node to begin with?
I mean if you need you should do so in an orderly fashion basically
excavate instances
or shut instances down manually, put the compute node in maintenance mode.
On rebooting compute node remove it from maintenance mode, turn on
instances or migrate them back to this compute node should you need.
Od delete them if you wish.

There is this nova option:
resume_guests_state_on_host_boot=true

But it doesn't delete or shutdown instances but rather turns them on
automatically once compute host resumes.
which might also work for you, probably not just mentioning it any way.

I don't know of an option to stop/delete instance on compute node shutdown.

Another option check maybe you could shelve suspended instance and then
later delete them.

Shelving stops the instance and takes a snapshot of it. Then depending on
the value of the shelved_offload_time config option, the instance is
deleted from the hypervisor (0), never deleted (-1), or deleted after some
period of time (> 0). Note that it's just destroying the backing instance
on the hypervisor, the actual instance in the nova database is not deleted.
Then you can later unshelve the instance:


This might help, but do not if you mess with kvm without updating Nova you
might be left haning else where :)
https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/




On Thu, Nov 2, 2017 at 9:03 AM, Chris  wrote:

> Hello,
>
> When we shut down a compute node the instances running on it get
> suspended. This generates some difficulties with some applications like
> RabbitMQ dont like to be suspended. Is there a way to change this behavior
> so that the running instances gets killed or shutdown instead?
>
> Thanks in advance.
>
> Cheers,
> Chris
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
>



-- 

Tzach Shefi

Senior Quality Engineer, RHCSA

Red Hat



tsh...@redaht.comM: +972-54-4701080 IM: tshefi

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] Compute Node shutdown how to prevent instance suspend

[Chris]

Hello,

When we shut down a compute node the instances running on it get 
suspended. This generates some difficulties with some applications like 
RabbitMQ dont like to be suspended. Is there a way to change this 
behavior so that the running instances gets killed or shutdown instead?


Thanks in advance.

Cheers,
Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack