Hi all, I'd like to get some discussion going on the subject of Django's middleware architecture. The current way middleware works provides a way for a developer to specify code executed at certain stages of the arrival of the request and the departure of the response from the system. This is done by the middleware class providing methods such as process_request, process_view, process_response etc.
Recently, I tried to write a piece of middleware to use the new Django transaction API (see django.db.transaction.atomic, a nice piece of restructuring by Aymeric Augustin). What I wanted was to have requests which modified data (request.method being one of POST, PUT, DELETE, PATCH) use a transaction, but read-only requests (GET, HEAD), which form the majority of a typical web application's requests, not to have to incur transaction overhead. Right away I hit the problem posed by the current Django middleware architecture. There is a *clear mismatch*. Using transactions in general, and also the new transaction.atomic API, implies wrapping the code concerned (the view function) with an envelope which starts the transaction on the way in, and commits the transaction (on success) or rolls it back (on failure) on the way out. This wrapping (like a try, except, finally) ensures that a transaction is not left open becauses of a failure to execute the except or finally block. The current Django middleware archictecture does not give this certainty. *The current Django middleware architecture cannot guarantee to always call a middleware's process_response or process_exception.* There are scenarios as mentioned in https://groups.google.com/d/msg/django-developers/U4-XcmvWLJ4/SwWBb24IyXUJwhere the required finalization code in a middleware's process_response of process_exception is not given a chance to run. Furthermore, the architecture *gives the appearance of being layered, but in fact is not.*If one of the middleware components short-circuits the process by returning a response from its process_request method, then all middleware classes' process_response methods will be called (unless an exception occurs) even deeper layer middleware classes whose process_request methods were not called. So, this need for transaction handling in middleware, and there are probably many other use cases which require a leak-free semantic, is highlighting a serious deficiency in the current Django middleware architecture. How can we address this? Try to imagine middleware components as layers around the eventual view function. The layers of an onion come to mind. Incoming requests pass through each layer on the way in, and responses or exceptions pass back out through the layers on the way out. We need a solution to prevent responses or exceptions skipping over layers on the way out if it passed through the layer on the way in. Wrappers provide a way to do this. If a middleware component *takes the form of a wrapper*, which wraps a view function (or a view function wrapped by another middleware wrapper), then that middleware can insert wrapping code to perform whatever it needs to do with the request and view_function when called (including generating a short-circuit response) and can perform whatever response or exception handling that is needed after calling the view function. This enables the try, except, finally block structure to be used, which prevents the leaking (or layer skipping) which the current Django middleware architecture is prone to. What would such a wrapper look like: def someMiddlewareWrapper(view_func): @wraps(view_func) def inner(request, *view_args, **view_kwargs): try: # Place code here which you would previously have # placed in a middleware class's process_request # or process_view method. If a short-circuit response # is generated, return it. # Call the view function: response = view_func(request, *args, **kwargs) # Place code here which you would previously have # placed in a middleware's process_response method. return response except Exception as e: # Place code here which you would previously have # placed in a middleware's process_exception method. # Re-raise the exception or return a response. finally: # Place any finalization code here. return inner How to hook this into Django: The developer would need to specify what the middleware wrappers are. This could be done in the settings module: MIDDLEWARE_WRAPPERS = [someMiddlewareWrapper] # List of middleware wrappers The Django request handling code (core/handlers/base.py get_response method) would take care of applying the wrappers when processing a request: (It currently uses "callback" to refer to the view function). # Apply wrappers in reverse so that requests pass through wrappers in the order they are listed for middleware_wrapper in reversed(settings.MIDDLEWARE_WRAPPERS): callback = middleware_wrapper(callback) and then invoke the view function: response = callback(request, *callback_args, **callback_kwargs) Control passes through the layers of middleware wrapping code, and eventually to the view function. The response makes its way outwards through the middleware wrapping code, with exceptions being handled where needed or allowed to propagate to outer layers. *What about existing middleware classes and backward compatability?* Can we somehow enable existing middleware classes to continue to exist, for a time, with a deprecation warning? It is possible to instantiate these classes (once only, as currently, but with a deprecation warning) and then (in base.py's get_response) pass the instance and the view function to an adapter which returns a wrapped view function which invokes the middleware instance's process_request, process_view, process_response etc. *Issues:* The biggest backward incompatibility issue I see relates to applications which use deferred rendering, where some middleware changes a response's template or rendering context. If anyone has suggestions for solutions to this, please share. One idea I have, which is only a partial solution, is to add a "context" attribute to the request object. Middleware wrappers can set context variables (and push new contexts) if required, so that the rendering done by the view function can take these into account. The RequestContext(request) would access them, giving lookup priority to contexts pushed by outer middleware layers. Maybe something similar could be done to specify templates. *Conclusion*: The current Django middleware architecture has a serious deficiency because a middleware's response or exception handling can be bypassed unexpectedly, thereby causing finalization code not to be executed. This can cause problems for middleware wanting to control transactions, release resources, etc. I have proposed an alternative, with a way to handle current middleware classes during a deprecation period. Certain use cases (deferred rendering) will be problematic and maybe good solution suggestions will be made for this, otherwise there will be a backward incompatibility for such cases. Comments, suggestions and ideas welcome. I'd like to guage what support there is in the Django community for this, to please post a reply. Regards, Stephen Brooks -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/17d7a886-b889-4fb8-af35-bcde28ff2d17%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
