Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state
Hi Operators, Could you please let me know your opinion on below scenario? It will help me to proceed my work. Thank you, Abhishek -Original Message- From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com] Sent: Tuesday, July 04, 2017 2:52 PM To: OpenStack Development Mailing List (not for usage questions); openstack-operat...@lists.openstack.org Subject: Re: [Openstack-operators] [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state Hi operators, I want to know how evacuation of resized instances is handled in real environment. For example if the vm is in resized state and if the compute host on which the vm is resized goes down, then how will operator evacuate the vm. One possible way is to reset that vm state to error and then evacuate it to new compute host. Please refer below scenario for reference: Scenario: = Pre-conditions: 1. Config option allow_resize_to_same_host is False. 2. Instance path is not mounted on shared storage. 3. Three compute nodes: "compute node A", "compute node B" and "compute node C" Steps: 1. Boot an instance on "compute node A". 2. User tries to resize the newly created instance and nova-scheduler selects "compute node B" as a destination node for resize. In this case nova creates a instance directory on destination "compute node B" and mark the instance directory which is present on the source "compute node A" as "*_resize". Note that the resize operation is yet not confirmed and "compute node B" goes down. 3. Reset instance state to ERROR as nova allows evacuation only if instance state is 'ACTIVE', 'STOPPED' or 'ERROR'. 4. Evacuate the instance to "compute node C" using target_host option. As a result, instance files which were on "compute node B" will be cleaned up after compute service on it is up again, but instance files which were on "compute node A" marked as "*_resize" will never be cleaned up. As of now there is no periodic task in nova to perform cleanup of these kinds of scenarios. Questions: 1. is this the only possible way of evacuating the resized instances in real world scenario? 2. If yes is there any way to cleanup unused (*_resize) instance files from the source compute node other than cleaning up it manually? 3. Should we add support of evacuating of resized instances in nova? Please let me know your opinions about the same. Thank you, Abhishek Kekane -Original Message- From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com] Sent: Thursday, June 29, 2017 5:57 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state Hi Chris, IMO we cannot perform auto-confirm as confirming or reverting is user's choice, whereas reverting is not possible as the node where the instance is resized is down. As suggested by you allowing this in nova require additional work. It is possible if we take power-state into consideration for evacuation operation, i.e. while evacuation if instance vmstate is resized and power-state is shutoff then we can stop that instance after evacuation. Thank you, Abhishek Kekane -Original Message- From: Chris Friesen [mailto:chris.frie...@windriver.com] Sent: Wednesday, June 28, 2017 8:54 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state On 06/28/2017 05:50 AM, Kekane, Abhishek wrote: > In masakari, we are setting an instance to an error state if the > vmstate is resized before evacuating it to a new host. Arguably the instance should be set to an error state as soon as you notice that the compute node is down. > Once an instance (which was in > resized state) is evacuated then it becomes active on the new host. > The main problem with this implementation from user’s point of view is > the instance goes into active state after evacuation, it should be in > stopped state if the prior action on the instance before resizing was > stop. In masakari, It’s possible to set the vm state to stopped state > after evacuation but for a short period the instance will go into the active > state which is unacceptable. That's a valid point, I think. > *Proposing changes to Nova:* > > In the current nova code, if the instance is in stopped state before > evacuation, then it remains in the stopped state after evacuation is > complete. On the similar lines, we are proposing nova should allow > instance to be evacuated in resized state and after evacuation the > instance should remain in stopped state if the prior action on the instance > is stopped before resizing. The current nova code looks at the vm_state to decide wh
Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state
Hi operators, I want to know how evacuation of resized instances is handled in real environment. For example if the vm is in resized state and if the compute host on which the vm is resized goes down, then how will operator evacuate the vm. One possible way is to reset that vm state to error and then evacuate it to new compute host. Please refer below scenario for reference: Scenario: = Pre-conditions: 1. Config option allow_resize_to_same_host is False. 2. Instance path is not mounted on shared storage. 3. Three compute nodes: "compute node A", "compute node B" and "compute node C" Steps: 1. Boot an instance on "compute node A". 2. User tries to resize the newly created instance and nova-scheduler selects "compute node B" as a destination node for resize. In this case nova creates a instance directory on destination "compute node B" and mark the instance directory which is present on the source "compute node A" as "*_resize". Note that the resize operation is yet not confirmed and "compute node B" goes down. 3. Reset instance state to ERROR as nova allows evacuation only if instance state is 'ACTIVE', 'STOPPED' or 'ERROR'. 4. Evacuate the instance to "compute node C" using target_host option. As a result, instance files which were on "compute node B" will be cleaned up after compute service on it is up again, but instance files which were on "compute node A" marked as "*_resize" will never be cleaned up. As of now there is no periodic task in nova to perform cleanup of these kinds of scenarios. Questions: 1. is this the only possible way of evacuating the resized instances in real world scenario? 2. If yes is there any way to cleanup unused (*_resize) instance files from the source compute node other than cleaning up it manually? 3. Should we add support of evacuating of resized instances in nova? Please let me know your opinions about the same. Thank you, Abhishek Kekane -Original Message- From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com] Sent: Thursday, June 29, 2017 5:57 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state Hi Chris, IMO we cannot perform auto-confirm as confirming or reverting is user's choice, whereas reverting is not possible as the node where the instance is resized is down. As suggested by you allowing this in nova require additional work. It is possible if we take power-state into consideration for evacuation operation, i.e. while evacuation if instance vmstate is resized and power-state is shutoff then we can stop that instance after evacuation. Thank you, Abhishek Kekane -Original Message- From: Chris Friesen [mailto:chris.frie...@windriver.com] Sent: Wednesday, June 28, 2017 8:54 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state On 06/28/2017 05:50 AM, Kekane, Abhishek wrote: > In masakari, we are setting an instance to an error state if the > vmstate is resized before evacuating it to a new host. Arguably the instance should be set to an error state as soon as you notice that the compute node is down. > Once an instance (which was in > resized state) is evacuated then it becomes active on the new host. > The main problem with this implementation from user’s point of view is > the instance goes into active state after evacuation, it should be in > stopped state if the prior action on the instance before resizing was > stop. In masakari, It’s possible to set the vm state to stopped state > after evacuation but for a short period the instance will go into the active > state which is unacceptable. That's a valid point, I think. > *Proposing changes to Nova:* > > In the current nova code, if the instance is in stopped state before > evacuation, then it remains in the stopped state after evacuation is > complete. On the similar lines, we are proposing nova should allow > instance to be evacuated in resized state and after evacuation the > instance should remain in stopped state if the prior action on the instance > is stopped before resizing. The current nova code looks at the vm_state to decide whether or not it's allowable to evacuate, and while "stopped" is a valid state to evacuate from "resized" is not. In your scenario it's both "stopped" *and* "resized" simultaneously, but there's no way to represent that in the vmstate so I think we'd have to check the power state, which would mean extending the check_instance_state() routine since it doesn't currently handle the power state. The trickier question is how to handle the &q
Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state
Hi Chris, IMO we cannot perform auto-confirm as confirming or reverting is user's choice, whereas reverting is not possible as the node where the instance is resized is down. As suggested by you allowing this in nova require additional work. It is possible if we take power-state into consideration for evacuation operation, i.e. while evacuation if instance vmstate is resized and power-state is shutoff then we can stop that instance after evacuation. Thank you, Abhishek Kekane -Original Message- From: Chris Friesen [mailto:chris.frie...@windriver.com] Sent: Wednesday, June 28, 2017 8:54 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state On 06/28/2017 05:50 AM, Kekane, Abhishek wrote: > In masakari, we are setting an instance to an error state if the > vmstate is resized before evacuating it to a new host. Arguably the instance should be set to an error state as soon as you notice that the compute node is down. > Once an instance (which was in > resized state) is evacuated then it becomes active on the new host. > The main problem with this implementation from user’s point of view is > the instance goes into active state after evacuation, it should be in > stopped state if the prior action on the instance before resizing was > stop. In masakari, It’s possible to set the vm state to stopped state > after evacuation but for a short period the instance will go into the active > state which is unacceptable. That's a valid point, I think. > *Proposing changes to Nova:* > > In the current nova code, if the instance is in stopped state before > evacuation, then it remains in the stopped state after evacuation is > complete. On the similar lines, we are proposing nova should allow > instance to be evacuated in resized state and after evacuation the > instance should remain in stopped state if the prior action on the instance > is stopped before resizing. The current nova code looks at the vm_state to decide whether or not it's allowable to evacuate, and while "stopped" is a valid state to evacuate from "resized" is not. In your scenario it's both "stopped" *and* "resized" simultaneously, but there's no way to represent that in the vmstate so I think we'd have to check the power state, which would mean extending the check_instance_state() routine since it doesn't currently handle the power state. The trickier question is how to handle the "resized" state...after evacuating an instance in the "resized" state should you be able to revert the resize? If so, how would that work in the case where the instance was resized on the same host originally and that host is no longer available? If not, then you'll end up with resources permanently reserved on the host the instance was on before the resize. I suppose one option would be to auto-confirm the resize in the case of a resize-to-same-host, but that'll be tricky to process with the host not available. Also, it should be noted that when rebuilding/evacuating a "stopped" instance the nova code just boots it up as normal and sets the vm_state to "active", then realizes that it's supposed to be stopped and sets the task_state to "powering_off" and goes down the normal path to stop the instance, eventually setting the vm_state to "stopped". So you're still going to end up with the same state transitions as what you have now, though the timing will probably be a bit tighter. If you really want a stopped instance to not actually start up on a rebuild/evacuate then that would be additional work. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state
On 06/28/2017 05:50 AM, Kekane, Abhishek wrote: In masakari, we are setting an instance to an error state if the vmstate is resized before evacuating it to a new host. Arguably the instance should be set to an error state as soon as you notice that the compute node is down. Once an instance (which was in resized state) is evacuated then it becomes active on the new host. The main problem with this implementation from user’s point of view is the instance goes into active state after evacuation, it should be in stopped state if the prior action on the instance before resizing was stop. In masakari, It’s possible to set the vm state to stopped state after evacuation but for a short period the instance will go into the active state which is unacceptable. That's a valid point, I think. *Proposing changes to Nova:* In the current nova code, if the instance is in stopped state before evacuation, then it remains in the stopped state after evacuation is complete. On the similar lines, we are proposing nova should allow instance to be evacuated in resized state and after evacuation the instance should remain in stopped state if the prior action on the instance is stopped before resizing. The current nova code looks at the vm_state to decide whether or not it's allowable to evacuate, and while "stopped" is a valid state to evacuate from "resized" is not. In your scenario it's both "stopped" *and* "resized" simultaneously, but there's no way to represent that in the vmstate so I think we'd have to check the power state, which would mean extending the check_instance_state() routine since it doesn't currently handle the power state. The trickier question is how to handle the "resized" state...after evacuating an instance in the "resized" state should you be able to revert the resize? If so, how would that work in the case where the instance was resized on the same host originally and that host is no longer available? If not, then you'll end up with resources permanently reserved on the host the instance was on before the resize. I suppose one option would be to auto-confirm the resize in the case of a resize-to-same-host, but that'll be tricky to process with the host not available. Also, it should be noted that when rebuilding/evacuating a "stopped" instance the nova code just boots it up as normal and sets the vm_state to "active", then realizes that it's supposed to be stopped and sets the task_state to "powering_off" and goes down the normal path to stop the instance, eventually setting the vm_state to "stopped". So you're still going to end up with the same state transitions as what you have now, though the timing will probably be a bit tighter. If you really want a stopped instance to not actually start up on a rebuild/evacuate then that would be additional work. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [masakari][nova] Allow evacuation of instances in resized state
Hi Devs, Masakari [1] provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically recovering the KVM-based Virtual Machine(VM)s from failure events such as VM process down, provisioning process down, and nova-compute host failure. It also provides API service to manage and control the automated rescue mechanism. Masakari use cases: --- 1. Allow evacuation of an instance which is in resized state. 2. If instance was in stopped state before resizing then after evacuation it should remain in stopped state instead of active state. Problem from Nova: -- Nova does not allow evacuation of an instance which is in resized state. Implementation in masakari: --- In masakari, we are setting an instance to an error state if the vmstate is resized before evacuating it to a new host. Once an instance (which was in resized state) is evacuated then it becomes active on the new host. The main problem with this implementation from user's point of view is the instance goes into active state after evacuation, it should be in stopped state if the prior action on the instance before resizing was stop. In masakari, It's possible to set the vm state to stopped state after evacuation but for a short period the instance will go into the active state which is unacceptable. Proposing changes to Nova: In the current nova code, if the instance is in stopped state before evacuation, then it remains in the stopped state after evacuation is complete. On the similar lines, we are proposing nova should allow instance to be evacuated in resized state and after evacuation the instance should remain in stopped state if the prior action on the instance is stopped before resizing. If the vmstate is resized, the old vmstate in instance_systemmetadata will be either active/stop. Based on this old vmstate, nova compute can decide whether to set the vm state to active or stopped during evacuation. Please let me know your opinion about allowing evacuation of instances in resized state in nova. [1] https://github.com/openstack/masakari Thank you, Abhishek Kekane __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
