Nevermind- I *finally* figured it out.. It was this bit: "for contact in contacts[startIndex:startIndex+100]:"
The contacts.count() in this case would steadily decrease as I changed the status of the objects to no longer fit the queryset criteria. So the startIndex value would get to the point where it would be more than the total number of records (about halfway through).. In other words, as my startIndex went up, the number of records went down, and at half way, the startIndex was no longer a valid offset. Since I realize this now, I'm just using the limit of 100, since it the queryset automatically removes objects no longer matching the query.. Seems more obvious now that I know what it's doing, but I thought once I assigned a queryset to a variable, the data in the query stayed, regardless of what changes I make to an object.. :\ Thanks, Matt On Apr 5, 1:02 pm, Matthew Riedel <[email protected]> wrote: > Ok, I'm having a bit of a weird issue with a medium-sized QuerySet > (~21k records). I have an app that syncs information from our LDAP > Directory and creates shared contacts in Google Apps for our domain. > Google puts a limit on 100 entries per batch, so I'm cycling through > these 21k records 100 at a time. > > Here's the relevant code: > > contacts = Contact.objects.filter(status__in=['up', 'cr', 'dl']) > contactsLog.info("Retrieved %s contacts." % contacts.count()) > startIndex = 0 > while startIndex <= contacts.count(): > contactsLog.info("Sending records %s through %s of %s." % > (startIndex, startIndex+100, contacts.count())) > contactsFeed = gdata.contacts.data.ContactsFeed() > for contact in contacts[startIndex:startIndex+100]: > <...do stuff...> > > It all starts fine- I log out how many contacts are found, etc. and > it's getting the correct number (21k) and starts churning through them > 100 at a time. However, when it gets to about 10,800 or so, the query > seems to stop working. I know this because I write the contents of > each batch out to a file. Around that mark, I start getting empty > batches. I don't get any errors or anything; the job continues but > just starts creating empty batches at that point. > > The job finishes "successfully", but obviously it didn't. If I turn > around and run that job again, it happily picks up where it left off > and then does another 5-6000 records or so, then does the same thing > (the QuerySet appears to be empty, even though there should be records > left).. And again, if I restart it, it happily keeps going. > > I've tried numerous ways to get this to work- I've tried using > Django's Pagination module, I've tried using the list() function > around the QuerySet call. Nothing seems to work. > > My hunch is it might be hitting some cache limit, or apache memory > limit, but I'm unsure of how to figure that out. Especially since the > job runs to completion, it just stops getting records back. > > I'm running Django 1.2.4, with Apache/2.2.17 (Unix) w/ mod_wsgi 3.3 on > Solaris 10 X64. > > I'd be greatly appreciative if someone can point me in the right > direction. Let me know if there's output from any configuration files > you want to see that would be helpful. > > Thanks in advance, > Matt -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
