On 09/21/2017 01:43 AM, David Demelier wrote:
Hello all,
It looks like parsemail.sh takes very long time to parse a large mail.
I have a 20MB mail file and running parsemail.sh on it takes forever
and starts eating all the memory and CPU time.
See after 3 minutes:
547 www 1 103 0 351M 239M CPU1 1 8:31 100.68%
python2.7
What can I do to debug more?
This isn't super helpful, but I've found a similar thing on my instance
and the issue seems to be when calling "parse_patch". It basically winds
up taking a really big buffer, splitting into an array of lines, and
building up another big buffer. My solution has always been too much of
a hack to propose here, but we work around it with something like:
# Don't try and process >5Mb patches, they flood the server
if len(patch) > 5000000:
raise MemoryError('patch too large to process: %d' % len(patch))
patch = parse_patch(patch)[0]
And we just don't bother tracking large patches.
_______________________________________________
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
