On 07/04/2017 05:21 AM, Kekane, Abhishek wrote:
Hi operators,
I want to know how evacuation of resized instances is handled in real
environment.
For example if the vm is in resized state and if the compute host on which the
vm is resized goes down, then how will operator evacuate the vm.
One possible way is to reset that vm state to error and then evacuate it to new
compute host.
Please refer below scenario for reference:
Scenario:
=========
Pre-conditions:
--------------------
1. Config option allow_resize_to_same_host is False.
2. Instance path is not mounted on shared storage.
3. Three compute nodes: "compute node A", "compute node B" and "compute node C"
Steps:
--------
1. Boot an instance on "compute node A".
2. User tries to resize the newly created instance and nova-scheduler selects
"compute node B" as a destination node for resize.
In this case nova creates a instance directory on destination "compute node B" and mark the
instance directory which is present on the source "compute node A" as "*_resize".
Note that the resize operation is yet not confirmed and "compute node B" goes
down.
3. Reset instance state to ERROR as nova allows evacuation only if instance
state is 'ACTIVE', 'STOPPED' or 'ERROR'.
I don't understand why you would do this, Abhishek. Why not REVERT the
resize operation (which would clean up the _resize directory and files
on the original host A) and then try the resize again?
Best,
-jay
4. Evacuate the instance to "compute node C" using target_host option.
As a result, instance files which were on "compute node B" will be cleaned up after compute
service on it is up again, but instance files which were on "compute node A" marked as
"*_resize" will never be cleaned up. As of now there is no periodic task in nova to perform cleanup
of these kinds of scenarios.
Questions:
1. is this the only possible way of evacuating the resized instances in real
world scenario?
2. If yes is there any way to cleanup unused (*_resize) instance files from the
source compute node other than cleaning up it manually?
3. Should we add support of evacuating of resized instances in nova?
Please let me know your opinions about the same.
Thank you,
Abhishek Kekane
-----Original Message-----
From: Kekane, Abhishek [mailto:[email protected]]
Sent: Thursday, June 29, 2017 5:57 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in
resized state
Hi Chris,
IMO we cannot perform auto-confirm as confirming or reverting is user's choice,
whereas reverting is not possible as the node where the instance is resized is
down.
As suggested by you allowing this in nova require additional work. It is
possible if we take power-state into consideration for evacuation operation,
i.e. while evacuation if instance vmstate is resized and power-state is shutoff
then we can stop that instance after evacuation.
Thank you,
Abhishek Kekane
-----Original Message-----
From: Chris Friesen [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 8:54 PM
To: [email protected]
Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in
resized state
On 06/28/2017 05:50 AM, Kekane, Abhishek wrote:
In masakari, we are setting an instance to an error state if the
vmstate is resized before evacuating it to a new host.
Arguably the instance should be set to an error state as soon as you notice
that the compute node is down.
Once an instance (which was in
resized state) is evacuated then it becomes active on the new host.
The main problem with this implementation from user’s point of view is
the instance goes into active state after evacuation, it should be in
stopped state if the prior action on the instance before resizing was
stop. In masakari, It’s possible to set the vm state to stopped state
after evacuation but for a short period the instance will go into the active
state which is unacceptable.
That's a valid point, I think.
*Proposing changes to Nova:*
In the current nova code, if the instance is in stopped state before
evacuation, then it remains in the stopped state after evacuation is
complete. On the similar lines, we are proposing nova should allow
instance to be evacuated in resized state and after evacuation the
instance should remain in stopped state if the prior action on the instance is
stopped before resizing.
The current nova code looks at the vm_state to decide whether or not it's allowable to evacuate, and while
"stopped" is a valid state to evacuate from "resized" is not. In your scenario it's both
"stopped" *and* "resized"
simultaneously, but there's no way to represent that in the vmstate so I think
we'd have to check the power state, which would mean extending the
check_instance_state() routine since it doesn't currently handle the power
state.
The trickier question is how to handle the "resized" state...after evacuating an instance
in the "resized" state should you be able to revert the resize? If so, how would that
work in the case where the instance was resized on the same host originally and that host is no
longer available? If not, then you'll end up with resources permanently reserved on the host the
instance was on before the resize. I suppose one option would be to auto-confirm the resize in the
case of a resize-to-same-host, but that'll be tricky to process with the host not available.
Also, it should be noted that when rebuilding/evacuating a "stopped" instance the nova code just boots it up
as normal and sets the vm_state to "active", then realizes that it's supposed to be stopped and sets the
task_state to "powering_off" and goes down the normal path to stop the instance, eventually setting the
vm_state to "stopped". So you're still going to end up with the same state transitions as what you have now,
though the timing will probably be a bit tighter. If you really want a stopped instance to not actually start up on a
rebuild/evacuate then that would be additional work.
Chris
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence for
the sole use of the addressee and may contain legally privileged, confidential,
and proprietary data. If you are not the intended recipient, please advise the
sender by replying promptly to this email and then delete and destroy this
email and any attachments without any further use, copying or forwarding.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
