On 25 October 2013 23:46, Day, Phil <philip....@hp.com> wrote:
> Hi Folks,
> We're very occasionally seeing problems where a thread processing a create 
> hangs (and we've seen when taking to Cinder and Glance).  Whilst those issues 
> need to be hunted down in their own rights, they do show up what seems to me 
> to be a weakness in the processing of delete requests that I'd like to get 
> some feedback on.
> Delete is the one operation that is allowed regardless of the Instance state 
> (since it's a one-way operation, and users should always be able to free up 
> their quota).   However when we get a create thread hung in one of these 
> states, the delete requests when they hit the manager will also block as they 
> are synchronized on the uuid.   Because the user making the delete request 
> doesn't see anything happen they tend to submit more delete requests.   The 
> Service is still up, so these go to the computer manager as well, and 
> eventually all of the threads will be waiting for the lock, and the compute 
> manager will stop consuming new messages.
> The problem isn't limited to deletes - although in most cases the change of 
> state in the API means that you have to keep making different calls to get 
> past the state checker logic to do it with an instance stuck in another 
> state.   Users also seem to be more impatient with deletes, as they are 
> trying to free up quota for other things.
> So while I know that we should never get a thread into a hung state into the 
> first place, I was wondering about one of the following approaches to address 
> just the delete case:
> i) Change the delete call on the manager so it doesn't wait for the uuid 
> lock.  Deletes should be coded so that they work regardless of the state of 
> the VM, and other actions should be able to cope with a delete being 
> performed from under them.  There is of course no guarantee that the delete 
> itself won't block as well.

I like this.

> ii) Record in the API server that a delete has been started (maybe enough to 
> use the task state being set to DELETEING in the API if we're sure this 
> doesn't get cleared), and add a periodic task in the compute manager to check 
> for and delete instances that are in a "DELETING" state for more than some 
> timeout. Then the API, knowing that the delete will be processes eventually 
> can just no-op any further delete requests.

There may be multiple API servers; global state in an API server seems
fraught with issues.

> iii) Add some hook into the ServiceGroup API so that the timer could depend 
> on getting a free thread from the compute manager pool (ie run some no-op 
> task) - so that of there are no free threads then the service becomes down. 
> That would (eventually) stop the scheduler from sending new requests to it, 
> and make deleted be processed in the API server but won't of course help with 
> commands for other instances on the same host.

This seems a little kludgy to me.

> iv) Move away from having a general topic and thread pool for all requests, 
> and start a listener on an instance specific topic for each running instance 
> on a host (leaving the general topic and pool just for creates and other 
> non-instance calls like the hypervisor API).   Then a blocked task would only 
> affect request for a specific instance.

That seems to suggest instance  # topics? Aieee. I don't think that
solves the problem anyway, because either a) you end up with a tonne
of threads, or b) you have a multiplexing thread with the same
potential issue.

You could more simply just have a dedicated thread pool for deletes,
and have no thread limit on the pool. Of course, this will fail when
you OOM :). You could do a dict with instance -> thread for deletes
instead, without creating lots of queues.

> I'm tending towards ii) as a simple and pragmatic solution in the near term, 
> although I like both iii) and iv) as being both generally good enhancments - 
> but iv) in particular feels like a pretty seismic change.

My inclination would be (i) - make deletes nonblocking idempotent with
lazy cleanup if resources take a while to tear down.


Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud

OpenStack-dev mailing list

Reply via email to