On 08/25/15 at 06:08pm, Gary Kotton wrote:
On 8/25/15, 9:10 AM, "Matt Riedemann" <[email protected]> wrote:
On 8/25/2015 10:03 AM, Gary Kotton wrote:
On 8/25/15, 7:04 AM, "Matt Riedemann" <[email protected]>
wrote:
On 8/24/2015 9:32 PM, Gary Kotton wrote:
In item #2 below the reboot is down via the guest and not the nova
api¹s :)
From: Gary Kotton <[email protected] <mailto:[email protected]>>
Reply-To: OpenStack List <[email protected]
<mailto:[email protected]>>
Date: Monday, August 24, 2015 at 7:18 PM
To: OpenStack List <[email protected]
<mailto:[email protected]>>
Subject: [openstack-dev] [nova] periodic task
Hi,
A couple of months ago I posted a patch for bug
https://launchpad.net/bugs/1463688. The issue is as follows: the
periodic task detects that the instance state does not match the state
on the hypervisor and it shuts down the running VM. There are a number
of ways that this may happen and I will try and explain:
1. Vmware driver example: a host where the instances are running
goes
down. This could be a power outage, host failure, etc. The first
iteration of the perdioc task will determine that the actual
instacne is down. This will update the state of the instance to
DOWN. The VC has the ability to do HA and it will start the
instance
up and running again. The next iteration of the periodic task
will
determine that the instance is up and the compute manager will
stop
the instance.
2. All drivers. The tenant decides to do a reboot of the instance
and
that coincides with the periodic task state validation. At this
point in time the instance will not be up and the compute node
will
update the state of the instance as DWON. Next iteration the
states
will differ and the instance will be shutdown
Basically the issue hit us with our CI and there was no CI running
for a
couple of hours due to the fact that the compute node decided to
shutdown the running instances. The hypervisor should be the source of
truth and it should not be the compute node that decides to shutdown
instances. I posted a patch to deal with this
https://review.openstack.org/#/c/190047/. Which is the reason for this
mail. The patch is backwards compatible so that the existing
deployments
and random shutdown continues as it works today and the admin now has
an
ability just to do a log if there is a inconsistency.
We do not want to disable the periodic task as knowing the current
state
of the instance is very important and has a ton of value, we just do
not
want the periodic to task to shut down a running instance.
Thanks
Gary
_______________________________________________________________________
__
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
In #2 the guest shouldn't be rebooted by the user (tenant) outside of
the nova-api. I'm not sure if it's actually formally documented in the
nova documentation, but from what I've always heard/known, nova is the
control plane and you should be doing everything with your instances
via
the nova-api. If the user rebooted via nova-api, the task_state would
be set and the periodic task would ignore the instance.
Matt, this is one case that I showed where the problem occurs. There are
others and I can invest time to see them. The fact that the periodic
task
is there is important. What I don¹t understand is why having an option
of
log indication for an admin is something that is not useful and instead
we
are going with having the compute node shutdown instance when this
should
not happen. Our infrastructure is behaving like cattle. That should not
be
the case and the hypervisor should be the source of truth.
This is a serious issue and instances in production can and will go
down.
--
Thanks,
Matt Riedemann
________________________________________________________________________
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_________________________________________________________________________
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
For the HA case #1, the periodic task checks to see if the instance.host
doesn't match the compute service host [1] and skips if they don't match.
Shouldn't your HA scenario be updating which host the instance is
running on? Or is this a vCenter-ism?
The nova compute node has not changed. It is not the compute nodes host.
The host that the instance was running on was down and those instances
were moved.
So this is a case where a single compute node is managing multiple
hypervisors? It sounds like there is an assumption being made in the
periodic task that doesn't hold true for the VMware driver, that a
request for the power state of an instance would fail if the host was
down. This may be a better fix here: to not sync the state if the host
is down.
For libvirt the same issues could happen if a process goes down and is
restarted (there may be some race conditions). But I am not familiar
enough with the ins and outs there. Just the fact that suggesting in some
cases that people disable the periodic task indicates that this too is an
issue.
But seriously, we need this and the change is non intrusive, configuarble
and backwards compatible. Honestly I see no reason why this is bing
blocked.
The change seems to be under discussion here because this is adding more
complexity to an already quite complex method. I believe the desire is
to find a model that simplifies, or at least doesn't add to the
complexity of, the way that syncs are handled.
[1]
http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#
n5871
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
