Jacob Kaplan-Moss napisał(a): >>> Can you share any hints on how to reduce the memory usage in such >>> situation? The underlying database structure is rather complicated and I >>> would like to not do all queries manually. >> At this level -- hundreds of thousands of objects per query -- I doubt >> that any ORM solution is going to deliver decent performance. > > That may be true, but in this case it's actually a bug in Django: > under some (many?) circumstances QuerySets load the entire result set > into memory even when used with an iterator. The good news is that > this has been fixed; the bad news is that the fix is on the > queryset-refactor branch, which likely won't merge to trunk for at > least a few more weeks, if not longer. Also note that values() won't > really help you, either: the overhead for a model instance isn't all > that big; the problem you're running into is simply the shear number > of results. > > In your situation, I'd do one of two things: either switch to the qsrf > branch if you're a living-on-the-edge kind of guy, or else look into > using an ObjectPaginator to churn through results in chunks.
Ah, thanks. The application is in production state, so I cann't use any other Django version we have installed on the machines. I'll try with ObjectPaginator, leaving raw SQL as last resort. Thank you all for the hints. -- Jarek Zgoda Skype: jzgoda | GTalk: [EMAIL PROTECTED] | voice: +48228430101 "We read Knuth so you don't have to." (Tim Peters) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
