#18251: multithreading deadlock in django.models.loading.get_apps
-------------------------------+--------------------
Reporter: harm | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.3
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
On a production site we have encountered deadlocks, which after fierce
investigation pointed to the django internals.
We use a deployment: apache/mod_wsgi
apache config:
{{{
WSGIDaemonProcess <name_here> processes=1 threads=10 umask=0002 display-
name=%{GROUP} stack-size=524288
}}}
== details of deadlock ==
During the bootstrap of django, so the first requests it handles, a
classic ABBA deadlock can occur.
This results in that django never boots, and that NO requests are handled.
Just restarting apache works, (hoping that the same deadlock does not
appear again).
description of deadlock
{{{
thread A thread B
request foo
import foo.views.py
import foo.models.py request 'bar'
import
bar.models.py (acquires import lock B)
bootstraps django
calls django.models.loading.get_apps()
self.write_lock.acquire() (lock A!)
load app apps
. . .
django.models.loading.get_apps()
import bar.models.py (takes import lock. lock B!)
self.write_lock.acquire() (lock A!)
<Blocked> <Blocked>
}}}
NB: this exact deadlock was '''actually''' seen in stracktraces. This is
not a guess.
== How to reproduce ==
its hard to reproduce, you need high volume traffic + multithreaded
deployement + a big enough application that different urls trigger
compelete different code paths in your project.
I am not able to produce a simple test project that demonstrates this
problem
We had a 100% reproducible setup, where the site would lock up (almost)
every time when we touched wsgi.py during peak hours of the day.
1. When you have a projects that is not multithreaded deployed -> this
problem does not happen
2. When your application has a limited set of urls/views so that all
initial calls follow same code paths -> this problem does not happen
3. When application has low traffic (the very first request can finish,
without other requests beeing made) -> this problem does not happen.
== Workaround ==
We use the attached {{{DjangoWSGIHandler}}} wrapper as a workaround. That
solved the problem 100 % for us.
Its a drop-in replacement for {{{ DjangoWSGIHandler }}} and effectively
disables multithreading the first few request (allowing django to
bootstrap properly), and only afther the first requests act multitheading
again.
Please consider if this workaround should be applied to django itself.
(Put the Lock HIGHER the chain, to the wsgi handler level)
== Fix ==
Some init locking should be adding to {{{ DjangoWSGIHandler }}} around
the complete request.
django.core.handlers.wsgi.WSGIHandler() has such lock around middleware
loading.
{{{
#!python
if self._request_middleware is None:
self.initLock.acquire()
}}}
Something similar should be around the complete request, not just
middleware loading.
But only the very first request.
--
Ticket URL: <https://code.djangoproject.com/ticket/18251>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en.
