I am using the experimental blobstore file API to write a CSV file
containing some event data. There's a lot of data so I'm batching up
the writes. My code is running in a backend so I have lots of time but
I'm running out of memory, and I don't understand why.
Here's the main part of the code:
from __future__ import with_statement
from google.appengine.api import files
q = Event.all()
events = q.fetch(50)
while events:
with files.open(blobname, 'a') as f:
buf = StringIO()
for event in events:
buf.write(event.id)
buf.write(',')
buf.write(`event.logged`)
buf.write(',')
buf.write(event.type)
buf.write(',')
buf.write(event.timestamp)
buf.write(',')
needAmpersand = False
for prop in event.dynamic_properties():
if needAmpersand:
buf.write('&')
needAmpersand = True
buf.write(prop + '=' + str(getattr(event, prop)))
buf.write('\n')
f.write(buf.getvalue())
buf.close()
events = q.fetch(50)
files.finalize(blobname)
When this runs, it makes it round the while events loop about 20 times
before the process aborts having used up more than 140 mb of memory
and blown the limit on the backend.
I note that the with f.open causes f.close to be called each time it
finishes the with clause, as f.close() is called by f.__exit__()
I appear to be leaking memory. Is the file API holding everything in
memory until the finalize or is there a leak in my code? I don't think
I can call finalize as after that I can't append to the database.
Help?
Thanks
Jules
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
