Hi ,
I want to use the email package to parse emails with attachments upto
1GB. However I find that python crashes with a Memory error traceback
while parsing the email with even a 300MB attachment at this point :
self._cur.set_payload(EMPTYSTRING.join(lines)) --> feedparser.py
I have the email contents in a file and the code is like ( on
python2.5, winxp ) :
self.msg = email.message_from_file(self.stream)
...
...
#Check if any attachments at all
if self.msg.get_content_maintype() != 'multipart':
print 'No attachments in message'
return
for part in self.msg.walk():
# multipart/* are just containers
if part.get_content_maintype() == 'multipart':
continue
is_attachment = part.get('Content-Disposition')
if is_attachment is None :
#body = part.get_payload(decode=True)
#print 'Body' , body
continue
filename = part.get_filename()
counter = 1
print 'Filename' , filename
if not filename:
filename = 'part-%03d%s' % (counter, 'bin')
counter += 1
att_path = os.path.join(detach_dir, filename)
#Check if its already there
if not os.path.isfile(att_path) :
fp = open(att_path, 'wb')
fp.write(part.get_payload(decode=True))
fp.close()
My machine has 2GB RAM so memory is not a problem and it seems python
tries to allocate a large memory chunk while doing a list
concatenation operation.
Also it seems that peak memory used for parsing and extracting the
attachment is three times the attachment size :
1) 2x used for parsing
2) 1x used for extracting it
The only way to fix this seems to be rewriting the parser to not load
the attachment into memory at all and maybe write it to a file , pass
the file pointer to set_payload and decode the attachment in small
chunks in get_payload instead of loading the entire file.
Subclass message to accept a file pointer in set_payload, etc...
Is there any other way to fix it , maybe compile python with some
flags to allow list concatenation to access a larger amount of memory.
Thanks,
Vijay
_______________________________________________
Email-SIG mailing list
Email-SIG@python.org
Your options:
http://mail.python.org/mailman/options/email-sig/archive%40mail-archive.com
