One thing that has bitten me more times than I care to remember is
forgetting to disable DEBUG in settings.py before processing large
datasets. The SQL queries build up in connection.queries until the
process dies. I ended up patching my django install to store queries
in a wrapper around buffer so that only the 200 most recent queries
were stored instead of ALL of them. Aside from making sure debugging
is off/disabled I end up stepping over in batches:
ids=[o["id"] for MyModel.objects.all().values("id")]
count=len(ids)
steps=1000
for n in range(steps,count+steps,steps):
lo,hi= n-steps, n
objects=MyModel.objects.filter(pk__in=ids[lo:hi])
function_that_processes_batch(objects)
I've imported data sets of around 80k like this, and the memory usage
is usually less than 200m.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
