Re: [PHP] free allocated memory: HOW ?
> Reading in the template should only happen once and take roughly as much > memory as the file is big. > > If the file is 20k, memory should go up roughly by that much. > > If it's going up a lot more, something is going wrong (either you're > doing something wrong or you're not just reading the template - you're > doing something else along with that). > i completely agree with you ;) i will check if the template is really the reason for the big jump with logging the memory usage each line. apart from that, i am still wondering why the memory usage grows in each loop for about 30 megs... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
Am Freitag, den 13.04.2007, 16:03 +1000 schrieb Chris:
> > ok, the complete workflow is a little bit complicated. we are using a
> > workflow engine and the newsletter generator is one step of three. the
> > first cleans the statistics data, the second generates the new data and
> > the third is the one which generates the mails.
>
> Put this in between each line and see where your memory is jumping the most:
>
> error_log('in file ' . __FILE__ . ' at line ' . __LINE__ . '; memory
> usage is ' . number_format((memory_get_usage() / 1024), 4) . ' kb');
>
> Once you've worked out the biggest jumps, fix them, then work on the
> next one and so on.
>
>
> It's going to be a lot quicker doing something like that over asking us
> to help you work out thousands of lines of code.
>
sure, i just posted the code because i was asked to ;)
i figured out the biggest jump (after reading the template) but could
not fix it yet -.-
thats why I asked a general question concerning php memory handling.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
It can if you're trying to process a borked image... I've had imagecreatefromjpeg() eat memory up to almost 50x the size of the image before finally deciding it can't handle it and crapping out. That was *after* running it through getimagesize() with no problem at all. okay - good point - but in this case the OP is reading in an html template file, just a string of 20.7Kb, what could go wrong there? this is probably one issue... the template is an HTML template containing images like http://www.brainguide.com/xxx"/>. How does PHP handle this stuff? I suppose that it is just a string and the image at the URL is not really read, or am I wrong? The newsletter does not have any attachements. Ed --PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
> Just post your source already.
>
ok, the complete workflow is a little bit complicated. we are using a
workflow engine and the newsletter generator is one step of three. the
first cleans the statistics data, the second generates the new data and
the third is the one which generates the mails.
here is the concerning method:
/**
* This method generates the monthly newsletter with the
statistics
block for the specified period.
*
* @param NewsletterValue $vo Initialized NewsletterValue object
* @param Integer $dateFrom Timestamp representing the start
date of
the period
* @param Integer $dateTo Timestamp representing the end date
of the
period
*/
private final function generateMonthlyNewsletter($vo, $dateFrom,
$dateTo){
$this->logger->debug("Generating monthly newsletter
with statistics
blocks.");
// initialize variable for the abonents
$abonents = null;
// read the newsletter template
$asset = $vo->getAsset();
$theData = self::readTemplate("../../../" .
$this->templatePath .
$asset->getFileName());
$this->logger->debug("successfully read template,
MEMORY USAGE: " .
memory_get_usage()/1048576 . " MB");
// iterate over the abonents and generate customized
newsletter
content
$counter = 0;
$start = 0;
$offset = self::PROCESS_ABONENTS;
// get the first newsletter abonents
$personAction = new PersonEPBAction();
$abonents = $personAction->findAbonents($start,
$offset);
$size = $personAction->countFoundRows();
$this->logger->debug("Found " . $size . "
abonents");
$loops = ceil($size/self::PROCESS_ABONENTS);
$this->logger->debug("Need " . $loops . " loops at " .
self::PROCESS_ABONENTS . " abonents per loop");
for($i = 0; $i < $loops; $i++){
$start = $i*self::PROCESS_ABONENTS;
$abonents = $personAction->findAbonents($start,
self::PROCESS_ABONENTS);
$this->logger->debug("Loop " . $i . ", MEMORY
USAGE: " .
memory_get_usage()/1048576 . " MB");
foreach($abonents as $person){
$this->logger->debug("Generating mail
for " .
$person->getPersonId() . ", " . $person->getFullName());
// replace custom person fields
$text =
self::replaceTemplateVars($theData, $person, $dateFrom,
$dateTo);
// add statistics
$text =
self::addStatisticsBlocks($text, $person);
// add the text to the
person_newsletter table
$lvo = new PersonNewsletterLightValue();
$lvo->setNewsletterIdFk($vo->getNewsletterId());
$lvo->setPersonIdFk($person->getPersonId());
$lvo->setMimeTypeIdFk(2);
// text/html TODO: refactor!
$lvo->setName("brainGuide AG"); //
TODO: refactor!
$lvo->setEmail("[EMAIL PROTECTED]"); //
TODO: refactor!
$lvo->setMailto($person->getEmailBusiness());
$lvo->setSubject($vo->getSubject());
$lvo->setText($text);
$lvo->setTstamp(time());
$lvo->setSent(0);
$lvo->setViewed(0);
self::savePersonNewsletter($lvo);
// unset variables
// unset($text, $lvo);
$text = null;
$lvo = null;
sleep(1);
}
}
}
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
> Arthur Erdös wrote: > > ok, its not really fast, but it runs on my development machine (AMD64, 4 > > GB of RAM, Ubuntu). mysql on the same machine, database with 175 tables > > and 2,3 GB data. that could probably slow down scripts, don't you agree? > > no. you have 4 F'ing gigs of ram on your desktop and it's a freakin' DB - > they are > supposed to handle *massive* ammounts of data ... you might consider > analysing the > queries you have and figuring out what an INDEX is for. I suppose the database is not the problem, the indexes and foreign keys are all set correctly. we have the mysql enterprise version and had a mysql consultant here some time ago and our database is well designed. (enterprise version only on webserver but anyway...) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
> don't OMFG me - I can't read your cmdline from here you know, > and I'm not the one with a completely borked script/system/whatever. > a big sorry again for the missunderstanding!! ;) I did not omfg YOU! I did omfg ME when I saw that the script eats much more memory as memory_get_usage() returned!! so please excuse me if my wording was wrong, i'm not a native english speaker oO Far from it! I am greatly thankfull for people like you and the community here! > php will not turn a 20.7Kb file into 400+ Megs of consumed memory - unless > your read > and store it in memory 2 odd times. > > which still leaves the question as to what is taking 125 Megs of RAM *before* > you > even start processing. > > > > > the read method does following: > > > > // read the newsletter template > > $fh = fopen($file, 'r'); > > $theData = fread($fh, filesize($file)); > > fclose($fh); > > return $theData; > > which can more succinctly be written as: > > return file_get_contents($file); > I'll try to change fread to file_get_contents()... anyhow, each generation of 100 mails eats approx 30 more megs :( > > > >>> the size of the template is 20.7 Kb > >> no way in hell that 20.7Kb turns into 270 Megs if you read the file into > >> php. > >> > >> are you experiencing severe load on the server whilst the script is > >> running? > >> and/or is the script actually very, very slow? > > > > well, the script is not very, very slow... it takes ~3-4 seconds to > > generate each newsletter (including database querys for the customized > > data) > > 3-4 seconds is dead slow if you ask me - a script like this should be capable > of > making the average mailserver go completely apeshit assuming you'd be mailing > the > newsletters out directly after creating them [rather than storing them in a > db] (i.e. > pumping out lots of emails a second] ... I know this because I have a qmail > server > that has trouble keeping up with my massmailer scripts. > no email goes out, the mails are stored in the database. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
ok, its not really fast, but it runs on my development machine (AMD64, 4 GB of RAM, Ubuntu). mysql on the same machine, database with 175 tables and 2,3 GB data. that could probably slow down scripts, don't you agree? > On Thu, 2007-04-12 at 17:59 +0200, Arthur Erdös wrote: > > > > > > are you experiencing severe load on the server whilst the script is > > > running? > > > and/or is the script actually very, very slow? > > > > well, the script is not very, very slow... it takes ~3-4 seconds to > > generate each newsletter (including database querys for the customized > > data) > > I would consider that slow :| > > Cheers, > Rob. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
> int memory_get_usage ( [bool $real_usage] ) > > Returns the amount of memory, in bytes, that's currently being allocated to > your PHP script. > > it returns the number of *bytes* !!! sure, and memory_get_usage()/1048576 returns the number of megs, or am i wrong? o_O > > > > >> exactly how much ram does the server in question have that you can > >> allow a single script to schlock up almost 2gigs ?? > > > > dev machine 4 GB, webserver 8 GB of RAM > > does running top in the cmdline confirm that the script is really eating > 100's of Megs? > OMFG, YES it does... top shows the script eating 658 megs right after reading the template (~125 megs before) the read method does following: // read the newsletter template $fh = fopen($file, 'r'); $theData = fread($fh, filesize($file)); fclose($fh); return $theData; > > > > the size of the template is 20.7 Kb > > no way in hell that 20.7Kb turns into 270 Megs if you read the file into php. > > are you experiencing severe load on the server whilst the script is running? > and/or is the script actually very, very slow? well, the script is not very, very slow... it takes ~3-4 seconds to generate each newsletter (including database querys for the customized data) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
hihi, sorry for that! The Ads come from our Ad-Seller (or how is this
called in english) and we have nearly no influence on what kind of ads
are delivered ;)
Am Donnerstag, den 12.04.2007, 16:18 +0200 schrieb Jochem Maas:
> [EMAIL PROTECTED] wrote:
> > Couple of questions come to mind...
> >
>
> ...
>
> > 4. Please tell us these 'newsletters' don't contain information on "male
> > enhancement products" or "govt seized puppies"..hah
>
> this spring to my mind as well - but given his domainname (brainguide), which
> points to a seemingly respectable
> site/company I think we are okay on that front. oh but I loved the way the HP
> ad on 'his' homepage almost ate all
> my RAM before firefox finally offered to kill the script that was going nuts
> ('because Flash was running slowly' ...
> yeah, not to mention the rest of my machine) ... seems the OP really has a
> problem with memory consumption ;-).
>
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
answering Jochem: > WTF - 386 megs for an html file??? somehow I doubt this is > the correct number ... are you reading the memory consumption > correctly? when the newsletter generation starts memory_get_usage() says ~96 MB in use. So approx 290 megs more are used after reading the 20.7 Kb template with fread()! I am reading the memory consumption with memory_get_usage(), so I dont know what should be wrong with that... > exactly how much ram does the server in question have that you can > allow a single script to schlock up almost 2gigs ?? dev machine 4 GB, webserver 8 GB of RAM > if the values are customized I can't fathom why you would generate > 100 emails per loop. sorry, my fault, I meant I am generating 100 emails "once", sleep(2), then the next 100, and so on... In my case (50xx mails - that results in 51 "loops" a 100 mails). answering TG: Couple of questions come to mind... 1. How big is this template that you're loading? Not sure why almost any HTML file would take up 300+ meg of memory when loaded into a variable. Are you loading images and other content in as well or just the HTML file? the size of the template is 20.7 Kb 2. Replacing the placeholders with actual data increases memory that much? Damn.. maybe I'm just ignorant of how PHP maintains things in memory, but that seems kind of nuts too 3. Do you have to store each newsletter in memory at each loop or can you send them out then move to the next one without storing them in an array or whatever you're doing? I write each newsletter to a mysql database then move to the next 4. Please tell us these 'newsletters' don't contain information on "male enhancement products" or "govt seized puppies"..hah No, they don't ;) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
I am using PHP version 5.1.2. on my dev machine. The same problem with PHP 5.2.1 on webserver. I am generating mails based on a HTML Template. After reading the template with fread() memory_get_usage() says ~386 MB in use... Then the placeholders in the read template are replaced by customized values within a loop. The size of the used memory grows permanently in each loop by approx 30 MB (100 newsletters are generated per loop). any ideas what i am doing wrong and where i [ab]use php? ^^ > Arthur Erdös wrote: > > Hello all, > > > > is there a way to free memory allocated by variables in PHP?? This is a > > very important issue concerning long running scripts... > > this is a recurrent problem - not much can be done about it AFAIK ... I'd > be very glad to be proved wrong. > > > > > I have a script that generates > 5000 Newsletters and when the script > > finishes it uses 1.8 GB (!!) of RAM. Although I am using unset() to > > clean up variables (tried with $var = null too). > > okay this does suggest your doing something else very wrong - I personally > have code that regularly generates upto 20,000 individualised newsletters > without > going anywhere near this ammount of memory - granted I've hit my memory limit > a couple of times and had to jack it up to 128Megs (in practice the script > grab just over > 64Megs when it's doing it's worst). > > what version of php are you [ab]using? > > > > > I've read sth about php not giving the allocated memory back to the OS > > until a script finishes, is that right?? Is it possible to free the > > memory "by hand"? > > > > Any suggestion is appreciated! > > > > thx in advance > > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
I am using PHP version 5.1.2. on my dev machine. The same problem with PHP 5.2.1 on webserver. I am generating mails based on a HTML Template. After reading the template with fread() memory_get_usage() says ~386 MB in use... Then the placeholders in the read template are replaced by customized values within a loop. The size of the used memory grows permanently in each loop by approx 30 MB (100 newsletters are generated per loop). any ideas what i am doing wrong and where i [ab]use php? ^^ Jochem Maas schrieb: Arthur Erdös wrote: Hello all, is there a way to free memory allocated by variables in PHP?? This is a very important issue concerning long running scripts... this is a recurrent problem - not much can be done about it AFAIK ... I'd be very glad to be proved wrong. I have a script that generates > 5000 Newsletters and when the script finishes it uses 1.8 GB (!!) of RAM. Although I am using unset() to clean up variables (tried with $var = null too). okay this does suggest your doing something else very wrong - I personally have code that regularly generates upto 20,000 individualised newsletters without going anywhere near this ammount of memory - granted I've hit my memory limit a couple of times and had to jack it up to 128Megs (in practice the script grab just over 64Megs when it's doing it's worst). what version of php are you [ab]using? I've read sth about php not giving the allocated memory back to the OS until a script finishes, is that right?? Is it possible to free the memory "by hand"? Any suggestion is appreciated! thx in advance -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] free allocated memory: HOW ?
I am using PHP version 5.1.2. on my dev machine. The same problem with PHP 5.2.1 on webserver. I am generating mails based on a HTML Template. If you look at the appended screenshot you'll see that after reading the template with fread() memory_get_usage() says ~386 MB in use... Then the placeholders in the read template are replaced by customized values within a loop. The size of the used memory grows permanently in each loop by approx 30 MB. any ideas what i am doing wrong and where i [ab]use php? ^^ Jochem Maas schrieb: Arthur Erdös wrote: Hello all, is there a way to free memory allocated by variables in PHP?? This is a very important issue concerning long running scripts... this is a recurrent problem - not much can be done about it AFAIK ... I'd be very glad to be proved wrong. I have a script that generates > 5000 Newsletters and when the script finishes it uses 1.8 GB (!!) of RAM. Although I am using unset() to clean up variables (tried with $var = null too). okay this does suggest your doing something else very wrong - I personally have code that regularly generates upto 20,000 individualised newsletters without going anywhere near this ammount of memory - granted I've hit my memory limit a couple of times and had to jack it up to 128Megs (in practice the script grab just over 64Megs when it's doing it's worst). what version of php are you [ab]using? I've read sth about php not giving the allocated memory back to the OS until a script finishes, is that right?? Is it possible to free the memory "by hand"? Any suggestion is appreciated! thx in advance -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] free allocated memory: HOW ?
Hello all, is there a way to free memory allocated by variables in PHP?? This is a very important issue concerning long running scripts... I have a script that generates > 5000 Newsletters and when the script finishes it uses 1.8 GB (!!) of RAM. Although I am using unset() to clean up variables (tried with $var = null too). I've read sth about php not giving the allocated memory back to the OS until a script finishes, is that right?? Is it possible to free the memory "by hand"? Any suggestion is appreciated! thx in advance -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
