If you ask me, a message queue sits quite firmly outside the realm of what a web framework should be responsible for providing. I think you are also underestimating the amount of effort and thought that goes into writing a robust and scalable message queue. For one simple example: what if Apache decides to kill or restart the Django process after a certain amount of requests have been processed (and I know that many people have their servers set to do this)? The threads which were are running asynchronous tasks would be killed as well, losing any tasks which hadn't yet been completed.
However, you do bring up a good point: being able to do things asynchronously is one of many key factors in being able to write truly scalable code. It's for that reason, I think, that Brian Rosner and James Tauber have been working on, not a re-implementation of a message queue, but instead a pluggable connector to communicate with more robust message queues in their django-engine project [1]. I haven't kept close tabs on that project, but I think it's far from ready for prime-time. If something like django-engine were to be more fully fleshed-out and made more robust, then I think it would make a much better case for inclusion in contrib, rather than try to turn django's signals into something they weren't meant to be. And I'm pretty sure Brian and James accept patches and idea contributions :) Anyway, that's just my 2c. -Eric Florenzano [1] http://code.google.com/p/django-engine/ On Jan 3, 11:46 pm, zvoase <crack...@gmail.com> wrote: > Hi Developers, > Basically, I wanted to ask for some devel's opinions on something. > When using the Django signals framework (which, by the way, is a very > good way of decoupling reusable apps whilst allowing for deep > integration), I sometimes find that a signal receiver does not need to > be called in synchronicity with the rest of the receivers. To > demonstrate, I'm going to explain a common scenario when it might be > useful to have *asynchronous* signal receivers, and why I feel there > should be a special provision made for them in the signals framework. > > Let's suppose I have a comments app, and I write a signal which is > 'fired' when a new comment is posted. Now let's assume I also write a > mail notification app which listens for this signal. When a comment is > posted on an object, the owner of the object is notified via e-mail. > For argument's sake, let's also assume that this app is *not* using a > mail queue which is flushed several times a day via a cron job - the > owner should be notified as soon as possible after the publication of > the comment. This means that the mail notifier's receiver function > must send the e-mail when called. This raises the issue that e-mail > may take a while (i.e. from a few seconds to a couple of minutes) to > send; it is after all operating over a network connection. This is > unacceptable, as we would ideally like to minimise latency in the HTTP > response. At the moment, the Signal class processes receivers in > serial; one must finish execution before the next may begin. It is > not, however, *critical* that this e-mail is sent before the next > receiver is processed; it may be put into the background in another > thread, and finish within a reasonable amount of time. > > An obvious solution to this problem is to use threading or forking, > thus placing the sending of the e-mail into the background. From my > receiver function I can do this by creating a new thread and starting > it, then returning after starting the thread without ensuring that the > sending of the e-mail has been completed. However, we run into a snag > whereby a thread is being created, started and left alone, without any > conceivable way of 'reigning it back in'. This could result in major > memory leaks or undetected infinite loops, and there is no obvious way > of creating a thread from within the receiver function *without* this > problem. > > So, taking the idea of threading but reworking it slightly, there is > another way to deal with the problem. If the Signal object were aware > that a particular receiver function (or a collection thereof) were to > be run in the background, then it could essentially start all of these > tasks in tandem and also create a controller, which could be passed > back to the sender of the signal. This controller will contain all the > thread instances, and might have a 'join' method which would wait > until they had all finished executing; an infinite loop or memory leak > would therefore be controllable, or at least detectable, as the > threads would be referenced within the scope of the sender. > > Applying this solution to the problem of e-mail sending, I would write > a function as before that sent an e-mail in the usual way. I would > then use the typical 'signal.connect' method to connect my receiver to > the signal, but pass in an 'async=True' keyword argument. Then, when a > comment is posted, a new e-mail-sending thread is created, along with > a thread controller (itself probably just a subclass of list with a > 'join' method, or something along those lines). This would send the e- > mail and finish execution, without holding up my view function where > I've added the comment. > > I hope I've made my rationale for this feature clear. I've sent it to > the mailing list first because I think it's something which could be > debated (at a good length). The inclusion of asynchronous signal > receivers will help Django's signals framework in becoming a mature > message-queueing solution, and I think that the use of signals is > invaluable to properly decoupling functionality whilst maintaining a > state of integration between a diverse sets of Django applications. > > Regards, > Zack --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
