Sorry for flood, i forgot p4: Prohibit stack deletion if the current stack state, status = (DELETE, IN PROGRESS). Raise not supported exception in heat engine. It is possible because stack state will be updated before deleting.
On Tue, Feb 10, 2015 at 2:04 PM, Kairat Kushaev <[email protected]> wrote: > Hi all, > During the analysis of the following bug: > https://bugs.launchpad.net/heat/+bug/1418878 > i figured out that orchestration engine doesn't work properly in some > cases. > The case is the following: > trying to delete the same stack with resources n times in series. > It might happen if the stack deleting takes much time and a user is sending > the second delete request again. > Orchestration engine behavior is the following: > 1) When first stack-delete command comes to heat service > it acquires the stack lock and sends delete request for resources > to other clients. > Unfortunately, the command does not start to delete resources from heat > db. > 2) At that time second stack-delete command for the same stack > comes to heat engine. It steals the stack lock, waits 0.2 (hard-coded > constant!) > sec to allow previous stack-delete command finish the operations (of > course, > the first didn't manage to finish deleting on time). After that engine > service starts > the deleting again: > - Request resources from heat DB (They exist!) > - Send requests for delete to other clients (They do not exist > because of > point 1). > Finally, we have stack in DELETE_FAILED state because the clients raise > exceptions during stack delete. > I have some proposals how to fix it: > p1) Make waiting time (0.2 sec) configurable. It allows to finish > stack-delete ops > before the second command starts deleting. From my point of view, it is > just > workaround because different stacks (and operations) took different time. > p2) Try to deny lock stealing if the current thread executes deleting. As > an option, > we can wait for the other thread if stack is deleting but it seems that it > is not possible > to analyze with the current solution. > p3) Just leave it as it is. IMO, the last solution. > Do you have any other proposals how to manage such kind of cases? > Perhaps there is exists more proper solution. > > Thank You, > Kairat Kushaev > > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
