Ok, I'm having a bit of a weird issue with a medium-sized QuerySet
(~21k records). I have an app that syncs information from our LDAP
Directory and creates shared contacts in Google Apps for our domain.
Google puts a limit on 100 entries per batch, so I'm cycling through
these 21k records 100 at a time.
Here's the relevant code:
contacts = Contact.objects.filter(status__in=['up', 'cr', 'dl'])
contactsLog.info("Retrieved %s contacts." % contacts.count())
startIndex = 0
while startIndex <= contacts.count():
contactsLog.info("Sending records %s through %s of %s." %
(startIndex, startIndex+100, contacts.count()))
contactsFeed = gdata.contacts.data.ContactsFeed()
for contact in contacts[startIndex:startIndex+100]:
<...do stuff...>
It all starts fine- I log out how many contacts are found, etc. and
it's getting the correct number (21k) and starts churning through them
100 at a time. However, when it gets to about 10,800 or so, the query
seems to stop working. I know this because I write the contents of
each batch out to a file. Around that mark, I start getting empty
batches. I don't get any errors or anything; the job continues but
just starts creating empty batches at that point.
The job finishes "successfully", but obviously it didn't. If I turn
around and run that job again, it happily picks up where it left off
and then does another 5-6000 records or so, then does the same thing
(the QuerySet appears to be empty, even though there should be records
left).. And again, if I restart it, it happily keeps going.
I've tried numerous ways to get this to work- I've tried using
Django's Pagination module, I've tried using the list() function
around the QuerySet call. Nothing seems to work.
My hunch is it might be hitting some cache limit, or apache memory
limit, but I'm unsure of how to figure that out. Especially since the
job runs to completion, it just stops getting records back.
I'm running Django 1.2.4, with Apache/2.2.17 (Unix) w/ mod_wsgi 3.3 on
Solaris 10 X64.
I'd be greatly appreciative if someone can point me in the right
direction. Let me know if there's output from any configuration files
you want to see that would be helpful.
Thanks in advance,
Matt
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
