Hi all, During the analysis of the following bug: https://bugs.launchpad.net/heat/+bug/1418878 i figured out that orchestration engine doesn't work properly in some cases. The case is the following: trying to delete the same stack with resources n times in series. It might happen if the stack deleting takes much time and a user is sending the second delete request again. Orchestration engine behavior is the following: 1) When first stack-delete command comes to heat service it acquires the stack lock and sends delete request for resources to other clients. Unfortunately, the command does not start to delete resources from heat db. 2) At that time second stack-delete command for the same stack comes to heat engine. It steals the stack lock, waits 0.2 (hard-coded constant!) sec to allow previous stack-delete command finish the operations (of course, the first didn't manage to finish deleting on time). After that engine service starts the deleting again: - Request resources from heat DB (They exist!) - Send requests for delete to other clients (They do not exist because of point 1). Finally, we have stack in DELETE_FAILED state because the clients raise exceptions during stack delete. I have some proposals how to fix it: p1) Make waiting time (0.2 sec) configurable. It allows to finish stack-delete ops before the second command starts deleting. From my point of view, it is just workaround because different stacks (and operations) took different time. p2) Try to deny lock stealing if the current thread executes deleting. As an option, we can wait for the other thread if stack is deleting but it seems that it is not possible to analyze with the current solution. p3) Just leave it as it is. IMO, the last solution. Do you have any other proposals how to manage such kind of cases? Perhaps there is exists more proper solution.
Thank You, Kairat Kushaev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
