On 6 oct, 18:46, puzzler <[EMAIL PROTECTED]> wrote: > Let's say I want to provide a web form, where the user enters in some > data. Then, when the user hits the submit button, it triggers a very > computationally intensive program to process that data that will take, > say, 4 or 5 minutes.
Is it 'wet-finger' estimation or is it based on any serious benchmark ? > The user is taken to a page that shows a little > "waiting" animation; and when the data-processing is complete, the > result is shown. I'm imagining that the best way to do this is to > have the intensive computation put the result in a database when done, > and the "waiting" page keeps polling the database to see whether the > result is there. Not necessarily - I mean, 'polling the database' - but you'll indeed need to have a way to know when the computation is done !-) > So here are my questions: > > If I write the computationally intensive function as a standard python > function, and call it normally, will my whole Django app essentially > "freeze" and stop serving other users while it processes this > computation. Mostly depends on how you deploy django, but "production" deployment usually implies either multthreading or server process fork - else, even serving static pages would rapidly freeze. But: > If so, do I instead use threads or separate os processes > to handle the computation? If the intensive computation consumes too > much memory, or otherwise encounters some sort of error, will my whole > Django server die? Which solution best isolates the server from this > kind of error? if your computation is that heavy, your best bet is to totally isolate it from your front-end server. IOW : design the whole thing to delegate computation to a distinct "computation-server". How you handle communication between the two (or more...) servers is up to you (from row socket with custom protocol to http). > Because this is computationally intensive, maybe Python isn't the best > way to write this program. Depends on the kind of computation... Pure Python, while not ridiculous for a hi-level dynamic language, doesn't shine when it comes to number-crunching, for sure - but then, there are dedicated C- coded libs like numpy, and possible JIT compiler like psyco. But before even starting to look at any of these solutions, you may want to make sure you correctly designed and implemented your computation. FWIW, some very simple design/coding change can make a very huge difference. You may want to read this thread on comp.lang.py: http://groups.google.com/group/comp.lang.python/browse_thread/thread/21aef9dec3fb5403/fbece20fb4b18f9e where a simple rewrite using the appropriate data structure and algorithm and a couple simple optimizations ended up in running 143 times faster (and, not posted but I did have an eye on it to, eating quite less memory too, which can make a *very* huge difference when you have several instances running concurrently - the difference between using real RAM or the swap... ) > If I want to write this program in another > language, does Django offer any way to invoke such a program, Django is just a Python framework, so the real question is "does Python offer any way to invoke such a program". And the answer is that Python offers many ways to skin this cat, from a braindead 'os.system("mycommand here") to C-coded lib binding, including any mmap or any networking solution. > and > would such a program have access to the database which is controlled > by Django and its database API? The database is not "controlled" by Django, it's "accessed" by Django. Any program in any language having bindings for your RDBMS can access it - granted appropriate rights. > Thanks! Any comments, or pointers to appropriate documentation, is > much appreciated. I'm afraid you failed to give enough informations about your "intensive computation" (ie: what kind of computation, what input data does it requires, what does it outputs, is it mostly IO bound or cpu bound or else, etc), nor about the expected load. According to these informations, answers can range from "just call the function directly" to "you'll obviously need a cluster" - which of course makes a big difference. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
