CA-33707: fix a nasty queueing deadlock found by stress testing.
In the API call paths for {clean,hard}_{shutdown,reboot} we used to hold the
per-VM mutex and then block in the vm_lifecycle_op queue. It was possible to
deadlock if an event was generated first (eg DevThread), the handler was
queued, the handler reaches the head of the queue and then deadlocks trying to
acquire the same per-VM lock.
Instead we only hold the per-VM lock when doing the actual domain destroy and
recreate operations. We ask domains to shutdown without any lock held: this
means we may interleave (eg) an API call VM.clean_shutdown with an internal
guest reboot. Conflicts are resolved in favour of the API calls.
We should never block with the per-VM mutex held.
Note that both the event thread and the main API call path both now use the
same queue 'domU_internal_shutdown'.
We also add a set of unit tests to quicktest to check that for every relevant
API call + every relevant domain shutdown + both possible codepaths
(synchronous API + asynchronous event thread) the result is as expected.
Signed-off-by: David Scott <[email protected]>
15 files changed, 540 insertions(+), 170 deletions(-)
ocaml/idl/api_errors.ml | 3
ocaml/idl/datamodel.ml | 7
ocaml/xapi/OMakefile | 2
ocaml/xapi/events.ml | 23 ++-
ocaml/xapi/quicktest.ml | 1
ocaml/xapi/quicktest_lifecycle.ml | 194 ++++++++++++++++++++++++++
ocaml/xapi/vmops.ml | 91 ++++++------
ocaml/xapi/xapi_fist.ml | 17 ++
ocaml/xapi/xapi_vm.ml | 273 ++++++++++++++++++++++++++-----------
ocaml/xapi/xapi_vm.mli | 5
ocaml/xapi/xapi_vm_migrate.ml | 15 +-
ocaml/xenops/domain.ml | 62 ++++----
ocaml/xenops/domain.mli | 5
ocaml/xenops/watch.ml | 5
ocaml/xenops/xenops.ml | 7
_______________________________________________
xen-api mailing list
[email protected]
http://lists.xensource.com/mailman/listinfo/xen-api
