Re: Iterating over very large queryset

2008-01-31 Thread Ben Ford
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I've had pretty good results with SQLAlchemy on large datasets, that might be a painless way to solve the problem.. Ben Jarek Zgoda wrote: > Jacob Kaplan-Moss napisał(a): > Can you share any hints on how to reduce the memory usage in such

Re: Iterating over very large queryset

2008-01-31 Thread Jarek Zgoda
Jacob Kaplan-Moss napisał(a): >>> Can you share any hints on how to reduce the memory usage in such >>> situation? The underlying database structure is rather complicated and I >>> would like to not do all queries manually. >> At this level -- hundreds of thousands of objects per query -- I

Re: Iterating over very large queryset

2008-01-30 Thread Jacob Kaplan-Moss
On 1/30/08, James Bennett <[EMAIL PROTECTED]> wrote: > On Jan 30, 2008 10:21 AM, Jarek Zgoda <[EMAIL PROTECTED]> wrote: > > Can you share any hints on how to reduce the memory usage in such > > situation? The underlying database structure is rather complicated and I > > would like to not do all

Re: Iterating over very large queryset

2008-01-30 Thread James Bennett
On Jan 30, 2008 10:21 AM, Jarek Zgoda <[EMAIL PROTECTED]> wrote: > Can you share any hints on how to reduce the memory usage in such > situation? The underlying database structure is rather complicated and I > would like to not do all queries manually. At this level -- hundreds of thousands of

Iterating over very large queryset

2008-01-30 Thread Jarek Zgoda
Hello, I'd like to get some hints on memory usage by python/django program. I have standalone script that populates solr index with django application data. The queryset I have to process has over 16 objects. No matter if I use iterator() or not, the script eats more and more memory, finally