On Tue, 17 Jul 2007, Chuck Hagenbuch wrote:
Quoting Andrew Morgan <[EMAIL PROTECTED]>:
A more useful question might be, "Why does it require so much memory
to download a folder and how can the memory footprint be reduced?"
Note - I can trigger this error on my system also with a mailbox
that is approximately 1/3 my php memory limit.
If someone wants to trace this a bit, that'd be great.
Can you give me some directions on how to get started tracing memory usage?
I've never traced a PHP application. :)
http://us.php.net/memory-get-usage
If that's not enough, xdebug is the next step (http://xdebug.org/).
That was easy!
I added some simple debug statements to print out the memory usage at
various steps in imp/lib/Folder.php. I printed out the memory usage after
each message was retrieved inside the 'for' loop, at the end of the loop,
and after the str_replace. Here were my results:
-----------------------------------------------------------------
Jul 17 13:25:40 webmail4 HORDE[1819]: [imp] Memory allocated: 7661556 [on
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]
... Many messages go by with memory usage increasing gradually according
to the size of the message ...
Jul 17 13:26:23 webmail4 HORDE[1819]: [imp] Memory allocated: 51898876 [on
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]
Jul 17 13:26:23 webmail4 HORDE[1819]: [imp] Memory allocated after loop:
51899260 [on line 610 of "/var/www-test/horde/imp/lib/Folder.php"]
-----------------------------------------------------------------
At this point it apparently was stopped for exceeding the php memory limit
(100MB on my system).
Here are the results for a smaller folder that does not exceed the memory
limit:
-----------------------------------------------------------------
Jul 17 13:33:15 webmail4 HORDE[2427]: [imp] Memory allocated: 7634344 [on
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]
... etc etc ...
Jul 17 13:33:17 webmail4 HORDE[2427]: [imp] Memory allocated: 18452336 [on
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]
Jul 17 13:33:17 webmail4 HORDE[2427]: [imp] Memory allocated after loop:
18452756 [on line 610 of "/var/www-test/horde/imp/lib/Folder.php"]
Jul 17 13:33:17 webmail4 HORDE[2427]: [imp] Memory allocated after
str_replace: 18453140 [on line 613 of "/var/www-test/horde/imp/lib/Folder.php"]
-----------------------------------------------------------------
I'm almost positive that the memory usage doubles temporarily for this
function call:
$body = str_replace("\r\n", "\n", $body);
However, I was able to avoid this memory usage penalty by calling
str_replace within the loop on each message, rather than the entire huge
string at the end. This also uses less memory overall. I suspect that
PHP memory routines don't "cleanup" memory very well when the size of a
variable is reduced as it is in the str_replace() call above.
I've attached a patch which implements this change.
However, it still fails if you try to download a zipped folder. Again,
this is probably caused by the temporary need for twice the memory in
order to make a working copy of the folder. Unfortunately I don't see any
way around that.
In general, you'll need to set the PHP memory limit to twice the size of
your largest folder download, plus some overhead (approximately 10-15MB).
Andy--- lib/Folder.php.orig 2007-01-02 05:54:56.000000000 -0800
+++ lib/Folder.php 2007-07-17 14:12:00.000000000 -0700
@@ -596,13 +596,12 @@
require a space in front of single digit days. */
$date = sprintf('%s %2s %s', date('D M', $h->udate),
date('j', $h->udate), date('H:i:s Y', $h->udate));
$body .= 'From ' . $from . ' ' . $date . "\n";
- $body .= imap_fetchheader($_SESSION['imp']['stream'], $i,
FT_PREFETCHTEXT);
- $body .= imap_body($_SESSION['imp']['stream'], $i,
FT_PEEK) . "\n";
+ $body .= str_replace("\r\n", "\n",
imap_fetchheader($_SESSION['imp']['stream'], $i, FT_PREFETCHTEXT));
+ $body .= str_replace("\r\n", "\n",
imap_body($_SESSION['imp']['stream'], $i, FT_PEEK) . "\n");
}
}
}
- $body = str_replace("\r\n", "\n", $body);
return $body;
}
--
IMP mailing list - Join the hunt: http://horde.org/bounties/#imp
Frequently Asked Questions: http://horde.org/faq/
To unsubscribe, mail: [EMAIL PROTECTED]
