Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-30 Thread Dan Genin
Punishing benign users as a defense against (potentially) malicious users sounds like a bad strategy. This should not be a zero-sum game. On 10/28/2013 02:49 PM, Joshua Harlow wrote: Sure, convergence model is great and likely how it has to be done. Its just a question of what is that

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-29 Thread John Garbutt
On 25 October 2013 23:23, Chris Behrens cbehr...@codestud.com wrote: On Oct 25, 2013, at 3:46 AM, Day, Phil philip@hp.com wrote: Hi Folks, We're very occasionally seeing problems where a thread processing a create hangs (and we've seen when taking to Cinder and Glance). Whilst those

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Day, Phil
I’d disagree that that – from a user perspective they should always be able to delete an Instance regardless of its state, and the delete should always work (or at least always appear to work to the user so that it no longer counts against their quota, and they are no longer charged for it)

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Joshua Harlow
Except I think the CAP theorem would say that u can't accurately give back there quota under thing like network partitions. If nova-compute and the message queue have a network partition then u can release there quota but can't actually delete there vms. I would actually prefer to not release

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Clint Byrum
Excerpts from Joshua Harlow's message of 2013-10-28 09:01:44 -0700: Except I think the CAP theorem would say that u can't accurately give back there quota under thing like network partitions. If nova-compute and the message queue have a network partition then u can release there quota but

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Joshua Harlow
I wish everything was so simple in distributed systems (like openstack) but there are real boundaries and limits to doing something like a kill -9 correctly while retaining the consistency of the resources in your cloud (any inconsistency costs someone $$$). Sent from my really tiny device...

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Chris Friesen
On 10/28/2013 10:30 AM, Joshua Harlow wrote: I wish everything was so simple in distributed systems (like openstack) but there are real boundaries and limits to doing something like a kill -9 correctly while retaining the consistency of the resources in your cloud (any inconsistency costs

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Joshua Harlow
But there is a difference here that I think needs to be clear. Releasing the resources from nova (in the current way its done) means another individual can take those resources and that causes inconsistencies (bad for deployer). I think we talked about how we can make this better by putting the

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Joshua Harlow
Sure, convergence model is great and likely how it has to be done. Its just a question of what is that convergence model :) I agree that its bad customer service to say 'yes u tried to delete it but I am charging u anyway' but I think the difference is that the user actually still has access to

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-28 Thread Chris Friesen
On 10/28/2013 12:01 PM, Joshua Harlow wrote: But there is a difference here that I think needs to be clear. Releasing the resources from nova (in the current way its done) means another individual can take those resources and that causes inconsistencies (bad for deployer). I think we talked

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-26 Thread Joshua Harlow
An idea that others and I are having for a similar use case in cinder (or it appears to be similar). If there was a well defined state machine/s in nova with well defined and managed transitions between states then it seems like this state machine could resume on failure as well as be

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-26 Thread Alex Glikson
+1 Regards, Alex Joshua Harlow harlo...@yahoo-inc.com wrote on 26/10/2013 09:29:03 AM: An idea that others and I are having for a similar use case in cinder (or it appears to be similar). If there was a well defined state machine/s in nova with well defined and managed transitions

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-26 Thread Abhishek Lahiri
Deletes should only be allowed when the vm is in a power off state. This will allow consistent state transition. Thanks Al On Oct 26, 2013, at 8:55 AM, Joshua Harlow harlo...@yahoo-inc.com wrote: I think I will try to have a unconference at the HK summit about ideas the cinder developers

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-26 Thread Joshua Harlow
Potentially, Although I think the lack of formalization and visibility (and the ability to easily change its transitions) into the state machine is at this point causing part of this pain. If the state machine was well defined (and adjustable - to a degree...) then you could imagine only

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-26 Thread Abhishek Lahiri
This is a bit off topic , but in general it seems to me that the state transitions as you said are not clearly defined for many openstack components. Is there any effort underway to define these? As the software gets bigger and bigger this will help both developers and operators. Thanks

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-26 Thread Joshua Harlow
There is at least 1 such effort being discussed in cinder, other projects I can not say. I am hoping to gain more traction there as taskflow[1] I think can provide (or help provide) a foundation to help here. Taskflow itself has a well defined state machine [2]. But it's in the end up to the

[openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-25 Thread Day, Phil
Hi Folks, We're very occasionally seeing problems where a thread processing a create hangs (and we've seen when taking to Cinder and Glance). Whilst those issues need to be hunted down in their own rights, they do show up what seems to me to be a weakness in the processing of delete requests

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-25 Thread Robert Collins
On 25 October 2013 23:46, Day, Phil philip@hp.com wrote: Hi Folks, We're very occasionally seeing problems where a thread processing a create hangs (and we've seen when taking to Cinder and Glance). Whilst those issues need to be hunted down in their own rights, they do show up what

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-25 Thread Day, Phil
There may be multiple API servers; global state in an API server seems fraught with issues. No, the state would be in the DB (it would either be a task_state of Deleteing or some new delete_stated_at timestamp I agree that i) is nice and simple - it just has the minor risks that the delete

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-25 Thread Clint Byrum
Excerpts from Day, Phil's message of 2013-10-25 03:46:01 -0700: Hi Folks, We're very occasionally seeing problems where a thread processing a create hangs (and we've seen when taking to Cinder and Glance). Whilst those issues need to be hunted down in their own rights, they do show up

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-25 Thread Day, Phil
-Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 25 October 2013 17:05 To: openstack-dev Subject: Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem Excerpts from Day,

Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem

2013-10-25 Thread Chris Behrens
On Oct 25, 2013, at 3:46 AM, Day, Phil philip@hp.com wrote: Hi Folks, We're very occasionally seeing problems where a thread processing a create hangs (and we've seen when taking to Cinder and Glance). Whilst those issues need to be hunted down in their own rights, they do show up