Re: [openstack-dev] [heat] heat delete woes in Juno
On 26/03/15 15:30, Georgy Okrokvertskhov wrote: I attached an example of the template which is hanging right now in my Juno environment. I believe it hangs because of floating ip stuff and the way how it is attached to a VM. It is autogenerated, so please don't be disturbed by strange resource names. There were a bunch of bug fixes required around this, due to the Neutron API not being designed for orchestration: https://bugs.launchpad.net/heat/+bug/1299259 https://bugs.launchpad.net/heat/+bug/1399699 https://bugs.launchpad.net/heat/+bug/1399702 All of those were eventually backported to stable/juno, so it may be worth checking that you have the latest stable release. If there's still issues then you may need to add dependencies to the template yourself - in particularly complex cases, Heat just can't get enough information to do the Right Thing. cheers, Zane. Thanks Gosha On Thu, Mar 26, 2015 at 12:15 PM, Ala Rezmerita ala.rezmer...@cloudwatt.com mailto:ala.rezmer...@cloudwatt.com wrote: Hi Matt I had similar problems with heat, and the work-around that i used is to abandon the stack (heat stack-abandon), and then I delete stack resources created one by one. Hope this helps. Ala Rezmerita Software Engineer || Cloudwatt M: (+33) 06 77 43 23 91 tel:%28%2B33%29%2006%2077%2043%2023%2091 Immeuble Etik 892 rue Yves Kermen 92100 Boulogne-Billancourt – France *De: *Matt Fischer m...@mattfischer.com mailto:m...@mattfischer.com *À: *openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org *Envoyé: *Jeudi 26 Mars 2015 19:17:08 *Objet: *[openstack-dev] [heat] heat delete woes in Juno Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heatdelete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com http://www.mirantis.com/ Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
Steve, I have two environments now, and although they have some other differences, one of them is Juno.2 and I've also been unable to repro it there. That env also lacks the same scale, load, and HA config so I was trying to figure out whether it was Juno.2 or not, so thanks for the extra data point. I am rolling out stack-abandon soon and will try to also get Juno.2 heat out soon. If I hit more examples, I'll file bugs. Thanks for the assistance all. On Sun, Mar 29, 2015 at 2:35 PM, Steve Baker sba...@redhat.com wrote: On 28/03/15 09:05, Matt Fischer wrote: Pavlo, Here is a link to one of the stacks. It is fairly simple just some routers/nets/subnets. The description is a bit odd perhaps, but legal. I've changed the template to not point at IPs at internal DNS. http://paste.ubuntu.com/10690759/ I've not been able to reproduce with this template running my local Juno 2014.2.2 or a recent devstack, but I'm sure you can. I created and deleted this in a loop about 5 times and it finally failed to delete on the last run. Now that it is stuck in DELETE_FAILED no amount of deleting will help. I'm concerned that a template this simple can get stuck like this. You should be able to delete the underlying resources using the neutron command, then delete the stack. I will have stack_abandon enabled next week as it just landed in Puppet: https://review.openstack.org/#/c/168157/ and will plan on trying that then. We've needed a series of workarounds for neutron resources as there are often implicit dependencies created which are not declared in REST create calls. Its likely you've discovered some more implicit dependencies which we need to handle. Could you please raise a bug [1] with the following? - the above pasted template - the heat event-list of the DELETE_FAILED stack - the heat stack-show of the DELETE_FAILED stack [1] https://bugs.launchpad.net/heat/+filebug On Thu, Mar 26, 2015 at 12:40 PM, Pavlo Shchelokovskyy pshchelokovs...@mirantis.com wrote: Hi Matt, if it would be feasible/appropriate, could you provide us with templates for stacks that show this behavior (try to get them with heat template-show stack-name-or-id)? This would help us to test and understand the problem better. And yes, just the day before I was contacted by one of my colleagues who seems to experience similar problems with Juno-based OpenStack deployment (though I did not had a chance to look through the issue yet). Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com On Thu, Mar 26, 2015 at 8:17 PM, Matt Fischer m...@mattfischer.com wrote: Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions)
Re: [openstack-dev] [heat] heat delete woes in Juno
On 28/03/15 09:05, Matt Fischer wrote: Pavlo, Here is a link to one of the stacks. It is fairly simple just some routers/nets/subnets. The description is a bit odd perhaps, but legal. I've changed the template to not point at IPs at internal DNS. http://paste.ubuntu.com/10690759/ I've not been able to reproduce with this template running my local Juno 2014.2.2 or a recent devstack, but I'm sure you can. I created and deleted this in a loop about 5 times and it finally failed to delete on the last run. Now that it is stuck in DELETE_FAILED no amount of deleting will help. I'm concerned that a template this simple can get stuck like this. You should be able to delete the underlying resources using the neutron command, then delete the stack. I will have stack_abandon enabled next week as it just landed in Puppet: https://review.openstack.org/#/c/168157/ and will plan on trying that then. We've needed a series of workarounds for neutron resources as there are often implicit dependencies created which are not declared in REST create calls. Its likely you've discovered some more implicit dependencies which we need to handle. Could you please raise a bug [1] with the following? - the above pasted template - the heat event-list of the DELETE_FAILED stack - the heat stack-show of the DELETE_FAILED stack [1] https://bugs.launchpad.net/heat/+filebug On Thu, Mar 26, 2015 at 12:40 PM, Pavlo Shchelokovskyy pshchelokovs...@mirantis.com mailto:pshchelokovs...@mirantis.com wrote: Hi Matt, if it would be feasible/appropriate, could you provide us with templates for stacks that show this behavior (try to get them with heat template-show stack-name-or-id)? This would help us to test and understand the problem better. And yes, just the day before I was contacted by one of my colleagues who seems to experience similar problems with Juno-based OpenStack deployment (though I did not had a chance to look through the issue yet). Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com http://www.mirantis.com On Thu, Mar 26, 2015 at 8:17 PM, Matt Fischer m...@mattfischer.com mailto:m...@mattfischer.com wrote: Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heatdelete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
Pavlo, Here is a link to one of the stacks. It is fairly simple just some routers/nets/subnets. The description is a bit odd perhaps, but legal. I've changed the template to not point at IPs at internal DNS. http://paste.ubuntu.com/10690759/ I created and deleted this in a loop about 5 times and it finally failed to delete on the last run. Now that it is stuck in DELETE_FAILED no amount of deleting will help. I'm concerned that a template this simple can get stuck like this. I will have stack_abandon enabled next week as it just landed in Puppet: https://review.openstack.org/#/c/168157/ and will plan on trying that then. On Thu, Mar 26, 2015 at 12:40 PM, Pavlo Shchelokovskyy pshchelokovs...@mirantis.com wrote: Hi Matt, if it would be feasible/appropriate, could you provide us with templates for stacks that show this behavior (try to get them with heat template-show stack-name-or-id)? This would help us to test and understand the problem better. And yes, just the day before I was contacted by one of my colleagues who seems to experience similar problems with Juno-based OpenStack deployment (though I did not had a chance to look through the issue yet). Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com On Thu, Mar 26, 2015 at 8:17 PM, Matt Fischer m...@mattfischer.com wrote: Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
I will try to gather up these templates and post them somewhere once I scrub any company specific stuff. Might be a day or so. Thanks! On Thu, Mar 26, 2015 at 12:40 PM, Pavlo Shchelokovskyy pshchelokovs...@mirantis.com wrote: Hi Matt, if it would be feasible/appropriate, could you provide us with templates for stacks that show this behavior (try to get them with heat template-show stack-name-or-id)? This would help us to test and understand the problem better. And yes, just the day before I was contacted by one of my colleagues who seems to experience similar problems with Juno-based OpenStack deployment (though I did not had a chance to look through the issue yet). Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com On Thu, Mar 26, 2015 at 8:17 PM, Matt Fischer m...@mattfischer.com wrote: Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
Regarding item #3: I have mainly seen this issue on stacks that have been snapshotted: https://bugs.launchpad.net/heat/+bug/1412965 In such cases, the only way to avoid (afaik) is for the owner to manually delete the snapshots prior to deleting the stack. Heat tries to auto-delete snapshots and hangs otherwise. On 03/26/2015 11:17 AM, Matt Fischer wrote: Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heatdelete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
On 26/03/15 15:56, Matt Fischer wrote: We don't have heat stack-abandon enabled. It's marked as a preview feature, have you had any issues? It's fairly safe for this use case. If it's important for you that you don't lose track of your resources and you want to adopt them again later, it's a bit more dicey: https://bugs.launchpad.net/heat/+bugs?field.tag=abandon-adopt Once you abandon the stack I assume you remove the resources outside of heat using nova/neutron etc? Correct. - ZB On Thu, Mar 26, 2015 at 1:15 PM, Ala Rezmerita ala.rezmer...@cloudwatt.com mailto:ala.rezmer...@cloudwatt.com wrote: Hi Matt I had similar problems with heat, and the work-around that i used is to abandon the stack (heat stack-abandon), and then I delete stack resources created one by one. Hope this helps. Ala Rezmerita Software Engineer || Cloudwatt M: (+33) 06 77 43 23 91 tel:%28%2B33%29%2006%2077%2043%2023%2091 Immeuble Etik 892 rue Yves Kermen 92100 Boulogne-Billancourt – France *De: *Matt Fischer m...@mattfischer.com mailto:m...@mattfischer.com *À: *openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org *Envoyé: *Jeudi 26 Mars 2015 19:17:08 *Objet: *[openstack-dev] [heat] heat delete woes in Juno Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heatdelete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
Hi Matt I had similar problems with heat, and the work-around that i used is to abandon the stack (heat stack-abandon), and then I delete stack resources created one by one. Hope this helps. Ala Rezmerita Software Engineer || Cloudwatt M: (+33) 06 77 43 23 91 Immeuble Etik 892 rue Yves Kermen 92100 Boulogne-Billancourt – France - Mail original - De: Matt Fischer m...@mattfischer.com À: openstack-dev@lists.openstack.org Envoyé: Jeudi 26 Mars 2015 19:17:08 Objet: [openstack-dev] [heat] heat delete woes in Juno Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete -stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
I attached an example of the template which is hanging right now in my Juno environment. I believe it hangs because of floating ip stuff and the way how it is attached to a VM. It is autogenerated, so please don't be disturbed by strange resource names. Thanks Gosha On Thu, Mar 26, 2015 at 12:15 PM, Ala Rezmerita ala.rezmer...@cloudwatt.com wrote: Hi Matt I had similar problems with heat, and the work-around that i used is to abandon the stack (heat stack-abandon), and then I delete stack resources created one by one. Hope this helps. Ala Rezmerita Software Engineer || Cloudwatt M: (+33) 06 77 43 23 91 Immeuble Etik 892 rue Yves Kermen 92100 Boulogne-Billancourt – France -- *De: *Matt Fischer m...@mattfischer.com *À: *openstack-dev@lists.openstack.org *Envoyé: *Jeudi 26 Mars 2015 19:17:08 *Objet: *[openstack-dev] [heat] heat delete woes in Juno Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 problematic-stack.hot Description: Binary data __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
We don't have heat stack-abandon enabled. It's marked as a preview feature, have you had any issues? Once you abandon the stack I assume you remove the resources outside of heat using nova/neutron etc? On Thu, Mar 26, 2015 at 1:15 PM, Ala Rezmerita ala.rezmer...@cloudwatt.com wrote: Hi Matt I had similar problems with heat, and the work-around that i used is to abandon the stack (heat stack-abandon), and then I delete stack resources created one by one. Hope this helps. Ala Rezmerita Software Engineer || Cloudwatt M: (+33) 06 77 43 23 91 Immeuble Etik 892 rue Yves Kermen 92100 Boulogne-Billancourt – France -- *De: *Matt Fischer m...@mattfischer.com *À: *openstack-dev@lists.openstack.org *Envoyé: *Jeudi 26 Mars 2015 19:17:08 *Objet: *[openstack-dev] [heat] heat delete woes in Juno Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat delete woes in Juno
Hi Matt, if it would be feasible/appropriate, could you provide us with templates for stacks that show this behavior (try to get them with heat template-show stack-name-or-id)? This would help us to test and understand the problem better. And yes, just the day before I was contacted by one of my colleagues who seems to experience similar problems with Juno-based OpenStack deployment (though I did not had a chance to look through the issue yet). Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com On Thu, Mar 26, 2015 at 8:17 PM, Matt Fischer m...@mattfischer.com wrote: Nobody on the operators list had any ideas on this, so re-posting here. We've been having some issues with heat delete-stack in Juno. The issues generally fall into three categories: 1) it takes multiple calls to heat to delete a stack. Presumably due to heat being unable to figure out the ordering on deletion and resources being in use. 2) undeleteable stacks. Stacks that refuse to delete, get stuck in DELETE_FAILED state. In this case, they show up in stack-list and stack-show, yet resource-list and stack-delete deny their existence. This means I can't be sure whether they have any real resources very easily. 3) As a corollary to item 1, stacks for which heat can never unwind the dependencies and stay in DELETE_IN_PROGRESS forever. Does anyone have any work-arounds for these or recommendations on cleanup? My main worry is removing a stack from the database that is still consuming the customer's resources. I also don't just want to remove stacks from the database and leave orphaned records in the DB. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
