Re: [PHP] memory allocation error
On Tue, Nov 13, 2012 at 12:23 AM, Carol Peck carolap...@gmail.com wrote: On 11/12/2012 11:51 AM, Matijn Woudt wrote: On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck carolap...@gmail.com wrote: Sebastian, Yes, I do , but this particular error never gets into my custom handler. I have also set it so that fatal errors fall through, and that doesn't seem to make any difference (again, probably because it never gets there). Carol Can you post the code of the error handler? I guess the bug is there. Bugs like these are mostly because of recursion errors or something. (Don't know if it's possible, but an error inside the error handler?) Fatal errors will btw always fall through, you can't catch fatal errors. You should, for testing, turn off your custom error handler and see what it does. - Matijn Ps. Please bottom-post on this mailing list. Here is the error class (I hope Im posting the code the right way). I will turn this off for a while. It's main purpose is to format the message before logging. I do know that things are getting logged properly. Thanks again. Snip some code Your hosting is using SuPHP which most likely causes these problems. You have two options: 1) Get a better host, or even better, get a dedicated server. 2) Increase your memory limit, even though your script doesn't really reaches that limit, it seems that SuPHP somehow messes that up. Read more at [1]. - Matijn [1] http://forum.inmotionhosting.com/viewtopic.php?t=3147
Re: [PHP] memory allocation error
On 11/13/2012 6:29 AM, Matijn Woudt wrote: On Tue, Nov 13, 2012 at 12:23 AM, Carol Peck carolap...@gmail.com mailto:carolap...@gmail.com wrote: On 11/12/2012 11:51 AM, Matijn Woudt wrote: On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck carolap...@gmail.com mailto:carolap...@gmail.com wrote: Sebastian, Yes, I do , but this particular error never gets into my custom handler. I have also set it so that fatal errors fall through, and that doesn't seem to make any difference (again, probably because it never gets there). Carol Can you post the code of the error handler? I guess the bug is there. Bugs like these are mostly because of recursion errors or something. (Don't know if it's possible, but an error inside the error handler?) Fatal errors will btw always fall through, you can't catch fatal errors. You should, for testing, turn off your custom error handler and see what it does. - Matijn Ps. Please bottom-post on this mailing list. Here is the error class (I hope Im posting the code the right way). I will turn this off for a while. It's main purpose is to format the message before logging. I do know that things are getting logged properly. Thanks again. Snip some code Your hosting is using SuPHP which most likely causes these problems. You have two options: 1) Get a better host, or even better, get a dedicated server. 2) Increase your memory limit, even though your script doesn't really reaches that limit, it seems that SuPHP somehow messes that up. Read more at [1]. - Matijn [1] http://forum.inmotionhosting.com/viewtopic.php?t=3147 Matijn, Thanks so much, I had found that post but thought it was related to the SilverStripe CMS they were referencing. I've been coming to the same conclusion about the host. Cheers, Carol
Re: [PHP] memory allocation error
Sebastian, Yes, I do , but this particular error never gets into my custom handler. I have also set it so that fatal errors fall through, and that doesn't seem to make any difference (again, probably because it never gets there). Carol On 11/11/2012 11:16 AM, Sebastian Krebs wrote: Hi, Do you use a custom error handler? Regards, Sebastian 2012/11/11 Carol Peckcarolap...@gmail.com Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing/html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
On 11/11/2012 08:45 AM, Carol Peck wrote: Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! Try adding exit or die at the end of what you know is the end of your scripts. See if the problem continues. Maybe at the very end of your customer error handler. If the problem stops showing up, you might want to look at your PHP config to see if anything is setup in the auto_append_file section. -- Jim Lucas http://www.cmsws.com/ http://www.cmsws.com/examples/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
Jim, Thanks for your idea - using die prevents it from coming up. As I mentioned, it is rather random so sometimes hard to verify. My auto_prepend and auto_append have no value in php.ini. I'm wondering why you suggested that? Best, Carol On 11/12/2012 8:09 AM, Jim Lucas wrote: On 11/11/2012 08:45 AM, Carol Peck wrote: Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! Try adding exit or die at the end of what you know is the end of your scripts. See if the problem continues. Maybe at the very end of your customer error handler. If the problem stops showing up, you might want to look at your PHP config to see if anything is setup in the auto_append_file section. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
Jim, I just found that the die didn't fix it after all - just ran into it again. So still looking for ideas! thanks, Carol On 11/12/2012 8:09 AM, Jim Lucas wrote: On 11/11/2012 08:45 AM, Carol Peck wrote: Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! Try adding exit or die at the end of what you know is the end of your scripts. See if the problem continues. Maybe at the very end of your customer error handler. If the problem stops showing up, you might want to look at your PHP config to see if anything is setup in the auto_append_file section. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
This could be an issue with the library you're using, adodb, I'd check to see if it has any debugging options to enable. I'm not familiar with it at all but that may be helpful. I'd also check out adodbs bug tracker, if one exists. Another suggestion would be use a profiler, such as xdebug, I believe thats the name. Carol Peck carolap...@gmail.com wrote: Jim, I just found that the die didn't fix it after all - just ran into it again. So still looking for ideas! thanks, Carol On 11/12/2012 8:09 AM, Jim Lucas wrote: On 11/11/2012 08:45 AM, Carol Peck wrote: Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! Try adding exit or die at the end of what you know is the end of your scripts. See if the problem continues. Maybe at the very end of your customer error handler. If the problem stops showing up, you might want to look at your PHP config to see if anything is setup in the auto_append_file section. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Re: Re: [PHP] memory allocation error
Original Message From: James ja...@nixsecurity.org To: Carol Peck carolap...@gmail.com, Jim Lucas li...@cmsws.com Cc: php-general@lists.php.net Sent: Mon, Nov 12, 2012, 12:53 PM Subject: Re: [PHP] memory allocation error This could be an issue with the library you're using, adodb, I'd check to see if it has any debugging options to enable. I'm not familiar with it at all but that may be helpful. I'd also check out adodbs bug tracker, if one exists. Another suggestion would be use a profiler, such as xdebug, I believe thats the name. Carol Peck carolap...@gmail.com wrote: Jim, I just found that the die didn't fix it after all - just ran into it again. So still looking for ideas! thanks, Carol On 11/12/2012 8:09 AM, Jim Lucas wrote: On 11/11/2012 08:45 AM, Carol Peck wrote: Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! Try adding exit or die at the end of what you know is the end of your scripts. See if the problem continues. Maybe at the very end of your customer error handler. If the problem stops showing up, you might want to look at your PHP config to see if anything is setup in the auto_append_file section. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Sorry for top posting. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
On 11/12/2012 7:50 AM, Carol Peck wrote: Jim, Thanks for your idea - using die prevents it from coming up. As I mentioned, it is rather random so sometimes hard to verify. My auto_prepend and auto_append have no value in php.ini. I'm wondering why you suggested that? If something was injecting code using the auto_append param, then you would not know about it, but it would still cause you issues. And by issuing a die or exit at the end of the code would show if it was your code or something running after all your script has completed. Best, Carol On 11/12/2012 8:09 AM, Jim Lucas wrote: On 11/11/2012 08:45 AM, Carol Peck wrote: Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! Try adding exit or die at the end of what you know is the end of your scripts. See if the problem continues. Maybe at the very end of your customer error handler. If the problem stops showing up, you might want to look at your PHP config to see if anything is setup in the auto_append_file section. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
On 11/12/2012 8:54 AM, Carol Peck wrote: Jim, I just found that the die didn't fix it after all - just ran into it again. So still looking for ideas! thanks, Carol Then it must be something in either your code or the way PHP is doing some garbage collection with the libs you are using. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation error
On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck carolap...@gmail.com wrote: Sebastian, Yes, I do , but this particular error never gets into my custom handler. I have also set it so that fatal errors fall through, and that doesn't seem to make any difference (again, probably because it never gets there). Carol Can you post the code of the error handler? I guess the bug is there. Bugs like these are mostly because of recursion errors or something. (Don't know if it's possible, but an error inside the error handler?) Fatal errors will btw always fall through, you can't catch fatal errors. You should, for testing, turn off your custom error handler and see what it does. - Matijn Ps. Please bottom-post on this mailing list.
Re: [PHP] memory allocation error
On 11/12/2012 11:51 AM, Matijn Woudt wrote:
On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck carolap...@gmail.com
mailto:carolap...@gmail.com wrote:
Sebastian,
Yes, I do , but this particular error never gets into my custom
handler.
I have also set it so that fatal errors fall through, and that
doesn't seem to make any difference (again, probably because it
never gets there).
Carol
Can you post the code of the error handler? I guess the bug is there.
Bugs like these are mostly because of recursion errors or something.
(Don't know if it's possible, but an error inside the error handler?)
Fatal errors will btw always fall through, you can't catch fatal
errors. You should, for testing, turn off your custom error handler
and see what it does.
- Matijn
Ps. Please bottom-post on this mailing list.
Here is the error class (I hope Im posting the code the right way). I
will turn this off for a while.
It's main purpose is to format the message before logging. I do know
that things are getting logged properly.
Thanks again.
?php
/**
* Error Handler class
*
* Handles E_USER_ERROR, E_USER_WARNING, E_WARNING, E_USER_NOTICE,
* and E_NOTICE errors.
*/
class ErrorHandler
{
/**
protected $_noticeLog = '/tmp/notice.log';
protected $_errorLog = '/tmp/error_handler.log';
public function __construct($message, $filename, $linenum, $vars)
{
$this-message = $message;
$this-filename = $filename;
$this-line = $linenum;
$this-vars = $vars;
}
public static function handle($errno, $errmsg, $filename, $line, $vars)
{
$self = new self($errmsg, $filename, $line, $vars);
switch ($errno) {
case E_USER_ERROR:
return $self-handleError();
case E_USER_WARNING:
case E_WARNING:
return $self-handleWarning();
case E_USER_NOTICE:
case E_NOTICE:
return $self-handleNotice();
default:
// Returning false tells PHP to revert control to the
// default error handler
return false;
}
}
public function handleError()
{
// Get backtrace
ob_start();
debug_print_backtrace();
$backtrace = ob_get_flush();
$body =EOT
A fatal error occured in the application:
Message: {$this-message}
File: {$this-filename}
Line: {$this-line}
Backtrace:
{$backtrace}
EOT;
// Use PHP's error_log() function to send an email
//error_log($body, 1, 'sysad...@example.com', Fatal error
occurred\n);
error_log($body, 1, 'cp...@wyom.net', Subject: Fatal error
occurred\n);
return error_log($body, 3, APP_PATH . $this-_errorLog);
exit(1); // non 0 exit status indicates script failure
}
/**
* Handle warnings
*
* Warnings indicate environmental errors; email sysadmin and
* continue
*
* @return boolean
*/
public function handleWarning()
{
$body =EOT
An environmental error occured in the application, and may indicate a
potential larger issue:
Message: {$this-message}
File: {$this-filename}
Line: {$this-line}
EOT;
$body = date('[Y-m-d H:i:s] ') . $body . \n;
return error_log($body,3, APP_PATH . $this-_errorLog);
}
/**
* Handle notices
*
* @return boolean
*/
public function handleNotice()
{
$body =EOT
A NOTICE was raised with the following information:
Message: {$this-message}
File: {$this-filename}
Line: {$this-line}
EOT;
$body = date('[Y-m-d H:i:s] ') . $body . \n;
return error_log($body, 3, APP_PATH . $this-_noticeLog);
}
}
/**
* Set ErrorHandler::handle() as default error handler
*/
set_error_handler(array('ErrorHandler', 'handle'));
[PHP] memory allocation error
Hi all, I've been chasing around a memory allocation error for some time and can't figure it out. It is somewhat random - I can run the script 3 times and then it will happen, or sometimes the first time. It happens at the very end of a script, actually after the script has finished running. I will see the following after the closing /html tag: Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 494142432 bytes) in Unknown on line 0 Just previous to this my mem usage is 3772104 The script itself does some database work (deletes, inserts, selects) but nothing very heavy and writes out an XML file. I use adodb 5.18, the server is PHP 5.3.18 (this was updated recently) and it is a VPS on inmotionhosting.com. As you can see, the memory limit is 256M so it's really high and I never see that I'm using more than 4M. The error doesn't fall through my error class - I'm assuming that's because it is happening after the script is complete. I am completely out of ideas on how to trap it or figure it out. Any ideas would be appreciated! -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] memory allocation
Dear list, I developed a script that have to create considerable array of integer numbers in memory. Currently my script has to accommodate in memory two arrays of around 120.000 numbers (for one year of data, I would like to reach a 2 years per query) that are loaded from a mysql server. I developed the script on a ubuntu 32bit machine and when I bring this script to our server, a 64bit opensuse the normal 128M of memory limit from php was exhausted. On my ubuntu system the script was consuming around 30Mb of memory / year, on the opensuse to run the same script it consumes more than 90Mb / year of memory. Is there a way to reduce this memory consumption on a 64bits machine ? Also, I made a test with four different virtual box machines (two opensuse and two ubuntu machines with the same PHP version, opensuse 11.4 and ubuntu 11.04 all running php-5.3.5) and it was quite choking the result for me. I am still wandering what is the difference between those and how to explain the results. A print screen of my machine running the four virtual boxes can be seen at: https://sites.google.com/site/foo4funreborn/phpcomp I would greatly thanks any help, with my best regards, Marcelo Bianchi -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation
On Tue, Sep 25, 2012 at 12:49 PM, Marcelo Bianchi mbian...@gfz-potsdam.de wrote: Dear list, I developed a script that have to create considerable array of integer numbers in memory. Currently my script has to accommodate in memory two arrays of around 120.000 numbers (for one year of data, I would like to reach a 2 years per query) that are loaded from a mysql server. I developed the script on a ubuntu 32bit machine and when I bring this script to our server, a 64bit opensuse the normal 128M of memory limit from php was exhausted. On my ubuntu system the script was consuming around 30Mb of memory / year, on the opensuse to run the same script it consumes more than 90Mb / year of memory. Is there a way to reduce this memory consumption on a 64bits machine ? Also, I made a test with four different virtual box machines (two opensuse and two ubuntu machines with the same PHP version, opensuse 11.4 and ubuntu 11.04 all running php-5.3.5) and it was quite choking the result for me. I am still wandering what is the difference between those and how to explain the results. A print screen of my machine running the four virtual boxes can be seen at: https://sites.google.com/site/foo4funreborn/phpcomp I would greatly thanks any help, with my best regards, Marcelo Bianchi Hi, If you suspect it's because of 64bits, you can always install 32bit PHP next to the 64bit version. At some point Ubuntu should support this out of the box with apt-get, but it seems that it's broken, atleast for now. Other than that, there's probably little you can do, as PHP will use 64bit integers and 64bit pointers internally, and there's pretty much nothing you can do about that. What I would suggest is just change the memory limit of PHP, you can do that with ini_set inside the PHP script. I assume you have plenty of ram in the system? If not, you should probably just go back to 32bit OS. - Matijn -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation
Hi Matijn, That is good test, to install php 32bit side to side with the 64bit to test. Will see how hard is to accomplish that. regards, Marcelo On 09/25/2012 01:58 PM, Matijn Woudt wrote: On Tue, Sep 25, 2012 at 12:49 PM, Marcelo Bianchi mbian...@gfz-potsdam.de wrote: Dear list, I developed a script that have to create considerable array of integer numbers in memory. Currently my script has to accommodate in memory two arrays of around 120.000 numbers (for one year of data, I would like to reach a 2 years per query) that are loaded from a mysql server. I developed the script on a ubuntu 32bit machine and when I bring this script to our server, a 64bit opensuse the normal 128M of memory limit from php was exhausted. On my ubuntu system the script was consuming around 30Mb of memory / year, on the opensuse to run the same script it consumes more than 90Mb / year of memory. Is there a way to reduce this memory consumption on a 64bits machine ? Also, I made a test with four different virtual box machines (two opensuse and two ubuntu machines with the same PHP version, opensuse 11.4 and ubuntu 11.04 all running php-5.3.5) and it was quite choking the result for me. I am still wandering what is the difference between those and how to explain the results. A print screen of my machine running the four virtual boxes can be seen at: https://sites.google.com/site/foo4funreborn/phpcomp I would greatly thanks any help, with my best regards, Marcelo Bianchi Hi, If you suspect it's because of 64bits, you can always install 32bit PHP next to the 64bit version. At some point Ubuntu should support this out of the box with apt-get, but it seems that it's broken, atleast for now. Other than that, there's probably little you can do, as PHP will use 64bit integers and 64bit pointers internally, and there's pretty much nothing you can do about that. What I would suggest is just change the memory limit of PHP, you can do that with ini_set inside the PHP script. I assume you have plenty of ram in the system? If not, you should probably just go back to 32bit OS. - Matijn -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation
Uhm, Okay,
I get it ... I am loading all at once, will try this out.
Here is what I do:
-- begin code
$network = IA;
$station = BJI;
$channel = BHZ;
$year = 2011;
$top = NULL;
$result = mysql_query('select '.$year.' as year, net, station, loc,
channel, start_day, start_time, end_day, end_time from data_'.$year.'
where net = '.$network.' and station = '.$station.' and scal_date
BETWEEN '.$year.'-01-01 AND '.$year.'-12-31 and channel =
'.$channel.' ') or die(mysql_error());
echo Loading $year ... ;
while ($r = mysql_fetch_array($result, MYSQL_ASSOC)) {
if ($top === NULL)
$top = new Segments($r, 120);
$top-add($r);
unset ($r);
}
echo Done\n;
echo Nothing free: .memory_get_usage(True). \n;
mysql_free_result($result);
mysql_close($db_link);
unset($result);
echo SQL free: .memory_get_usage(True). \n;
unset($top);
echo Segments free: .memory_get_usage(True). \n;
-- end code
The Segments object is a class with some functionality but, after
processing the results of each row it stores two integers (or floats due
to the large numbers, number of seconds from epoch * 1.0 +
microseconds).
I will try to implement this chunk mode of fetching the data from sql to
see if the php consumption goes down.
regards,
Marcelo
On 09/25/2012 02:23 PM, shiplu wrote:
I would like to see how you are reading data from database?
There are many ways to reduce memory usage. Here is a very common way
reduce memory.
instead of
$res = mysql_query(select * from table1 limit 1000);
while($row = mysql_fetch_assoc($res)){
/// process row.
}
Do this,
for($x = 0; $x 1000; $x+=100;}
$res = mysql_query(sprintf(select required_col1, required_col2,
... from table1 limit %d, %d,$x, 100));
while($row = mysql_fetch_assoc($res)){
/// process row.
}
}
--
Shiplu.Mokadd.im http://Shiplu.Mokadd.im
ImgSign.com | A dynamic signature machine
Innovation distinguishes between follower and leader
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation
I would like to see how you are reading data from database?
There are many ways to reduce memory usage. Here is a very common way
reduce memory.
instead of
$res = mysql_query(select * from table1 limit 1000);
while($row = mysql_fetch_assoc($res)){
/// process row.
}
Do this,
for($x = 0; $x 1000; $x+=100;}
$res = mysql_query(sprintf(select required_col1, required_col2,
... from table1 limit %d, %d,$x, 100));
while($row = mysql_fetch_assoc($res)){
/// process row.
}
}
--
Shiplu.Mokadd.im
ImgSign.com | A dynamic signature machine
Innovation distinguishes between follower and leader
Re: [PHP] memory allocation
On 09/25/2012 02:40 PM, shiplu wrote: Also if the final data on presentation layer is very small you can calculate it in mysql using stored procedure. okay, also something to try. thank you ! regards, marcelo -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Memory Allocation Error
[EMAIL PROTECTED] wrote:
Hi!
I have a script that reads a 120 MB remote file. This raises a Memory
Allocation Error unless I use:
ini_set('memory_limit', '130M');
I doubt this is good for my server... I tried both fopen and
file_get_contents. This used to work fine in PHP 4 until I upgraded to
PHP 5.
Any ideas?
If you need to keep a 120Mb file in core at any point in time, you'll
need the memory. Either adjust the memory-limit, or change your code
to process the file in chunks/lines/bytes.
/Per Jessen, Zürich
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Memory Allocation Error
Thank you! That works.
On Nov 6, 2007 12:23 AM, Per Jessen [EMAIL PROTECTED] wrote:
[EMAIL PROTECTED] wrote:
Hi!
I have a script that reads a 120 MB remote file. This raises a Memory
Allocation Error unless I use:
ini_set('memory_limit', '130M');
I doubt this is good for my server... I tried both fopen and
file_get_contents. This used to work fine in PHP 4 until I upgraded to
PHP 5.
Any ideas?
If you need to keep a 120Mb file in core at any point in time, you'll
need the memory. Either adjust the memory-limit, or change your code
to process the file in chunks/lines/bytes.
/Per Jessen, Zürich
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Memory Allocation Error
Hi!
I have a script that reads a 120 MB remote file. This raises a Memory
Allocation Error unless I use:
ini_set('memory_limit', '130M');
I doubt this is good for my server... I tried both fopen and
file_get_contents. This used to work fine in PHP 4 until I upgraded to
PHP 5.
Any ideas?
Thanks.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] memory allocation problems with fgets
Hi,
I have a script parsing the lines of a (rather large) file:
while (!feof($outPluginfh)) {
$outPlugin = fgets ($outPluginfh,1024);
# process $outPlugin here...
}
Now, with large files I was running into an memory allocation error
(after 16MB of allocation..).
It looks that php is allocating space for each fgets call, an not
releasing it, in spite of the return value being always assigned to the
same variable.
I red in other posts that for removing this 16MB limit I can simply put
the statement
ini_set(memory_limit,-1);
at the beginning, and in fact this line does the job.
However, I'm not happy because I lost control of how much memeory php
is allcating on my system, and I fear that with multiple users
triggering this script I'll sooner or later run into (worse) problems.
Is there any other more controllable way to do?
Thanks,
Maurizio Molina
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory allocation problems with fgets
# [EMAIL PROTECTED] / 2007-01-15 16:43:50 +:
Hi,
I have a script parsing the lines of a (rather large) file:
while (!feof($outPluginfh)) {
$outPlugin = fgets ($outPluginfh,1024);
# process $outPlugin here...
}
Now, with large files I was running into an memory allocation error
(after 16MB of allocation..).
It looks that php is allocating space for each fgets call, an not
releasing it, in spite of the return value being always assigned to the
same variable.
I red in other posts that for removing this 16MB limit I can simply put
the statement
ini_set(memory_limit,-1);
at the beginning, and in fact this line does the job.
What makes you sure that loop is the culprit? Does the script still
work if you set memory_limit to -1 right before the loop, and back to
its old value right after it?
--
How many Vietnam vets does it take to screw in a light bulb?
You don't know, man. You don't KNOW.
Cause you weren't THERE. http://bash.org/?255991
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] PHP Memory Allocation (checked via TOP)
Hi, I have written a simple test program to see how php allocates memory. Test code allocates ~10 Meg of RAM in an array within a loop till it runs out of memory as: $str = rand(65, 95) . rand(65, 95) . rand(65, 95) . rand(65, 95) . rand(65, 95); $aa[] = str_repeat($str, 200); What I don't understand here is, for every 10 Meg memory it allocates, mem usage goes up about 19 Megs when I look at this via top. Basically, it allocates from physical memory in the beginning, starts using swap when it is out of RES/Physical memory, which makes me assume that garbage collection should kick in for temporary string creations, and each new allocation increases the allocated memory size by 19 Megs. Any idea why this is happening, and why do you think memory allocation is so expensive in PHP? Thanks.. Results of TOP: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 21843 apache17 0 19292 5232 3832 S 0.0 0.5 0:00.04 php 21843 apache16 0 38824 24m 3908 S 0.0 2.4 0:00.10 php 21843 apache15 0 58356 43m 3912 S 0.0 4.3 0:00.17 php 21843 apache16 0 77888 62m 3912 S 0.0 6.2 0:00.22 php 21843 apache15 0 97420 81m 3912 S 0.0 8.1 0:00.29 php 21843 apache15 0 114m 100m 3912 S 0.0 9.9 0:00.35 php Results of free -m: total used free sharedbuffers cached Mem: 1011138872 0 2 58 Swap: 2008388 1619 Mem: 1011158852 0 2 58 Swap: 2008388 1619 Mem: 1011177833 0 2 58 Swap: 2008388 1619 Mem: 1011196814 0 2 58 Swap: 2008388 1619 Mem: 1011216795 0 2 58 Swap: 2008388 1619 Mem: 1011234776 0 2 58 Swap: 2008388 1619 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: PHP Memory Allocation (checked via TOP)
As a side note, even though user is shown as apache, this code was executed from command line. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Memory Allocation (checked via TOP)
Cabbar Duzayak wrote: Hi, I have written a simple test program to see how php allocates memory. Test code allocates ~10 Meg of RAM in an array within a loop till it runs out of memory as: $str = rand(65, 95) . rand(65, 95) . rand(65, 95) . rand(65, 95) . rand(65, 95); $aa[] = str_repeat($str, 200); What I don't understand here is, for every 10 Meg memory it allocates, mem usage goes up about 19 Megs when I look at this via top. Just looked over this really quickly, so I might have missed something, but it seems to me that you're allocating almost 20 megs... your $str is 10 bytes long (plus terminator and PHP overhead, whatever that turns out to be...), and you're repeating it two million times. 10 bytes times 2 million is 20 million bytes. 20 000 000 divided by (1024*1024) is 19.07 - it seems to me that you're allocating 19.07 megs, and top is showing you that it's allocating 19 megs...? Seems about right. jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP Memory Allocation (checked via TOP)
$str is 10 bytes then you repeat it 200 times That gives you 2000 bytes. That's 20M not 10M -Rasmus Cabbar Duzayak wrote: Hi, I have written a simple test program to see how php allocates memory. Test code allocates ~10 Meg of RAM in an array within a loop till it runs out of memory as: $str = rand(65, 95) . rand(65, 95) . rand(65, 95) . rand(65, 95) . rand(65, 95); $aa[] = str_repeat($str, 200); What I don't understand here is, for every 10 Meg memory it allocates, mem usage goes up about 19 Megs when I look at this via top. Basically, it allocates from physical memory in the beginning, starts using swap when it is out of RES/Physical memory, which makes me assume that garbage collection should kick in for temporary string creations, and each new allocation increases the allocated memory size by 19 Megs. Any idea why this is happening, and why do you think memory allocation is so expensive in PHP? Thanks.. Results of TOP: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 21843 apache17 0 19292 5232 3832 S 0.0 0.5 0:00.04 php 21843 apache16 0 38824 24m 3908 S 0.0 2.4 0:00.10 php 21843 apache15 0 58356 43m 3912 S 0.0 4.3 0:00.17 php 21843 apache16 0 77888 62m 3912 S 0.0 6.2 0:00.22 php 21843 apache15 0 97420 81m 3912 S 0.0 8.1 0:00.29 php 21843 apache15 0 114m 100m 3912 S 0.0 9.9 0:00.35 php Results of free -m: total used free sharedbuffers cached Mem: 1011138872 0 2 58 Swap: 2008388 1619 Mem: 1011158852 0 2 58 Swap: 2008388 1619 Mem: 1011177833 0 2 58 Swap: 2008388 1619 Mem: 1011196814 0 2 58 Swap: 2008388 1619 Mem: 1011216795 0 2 58 Swap: 2008388 1619 Mem: 1011234776 0 2 58 Swap: 2008388 1619 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] memory allocation in $var...
By doing this:
?php
$someArray=array("this","that","those");
for(something to loop)
do something with $someArray here;
?
is PHP using the same amount of memory as this:
?php
$someArray=array("this","that","those");
somefunction($someArray);
function somefunction($ref_to_array)
{
for(something to loop)
do something with $ref_to_array here;
}
?
because it seems to me, that the script gets a little slower when I
introduce the function method.
. Christian Dechery (lemming)
. http://www.tanamesa.com.br
. Gaita-L Owner / Web Developer
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP] memory allocation in $var...
I got the same results from my benchmarks, also.
Unless you want to change variable in function and get modified value, do not
use reference.
Without reference, PHP4 executes script faster. Due to reference counting in
PHP4, probably.
Regards,
--
Yasuo Ohgaki
"Christian Dechery" [EMAIL PROTECTED] wrote in message
[EMAIL PROTECTED]">news:[EMAIL PROTECTED]...
By doing this:
?php
$someArray=array("this","that","those");
for(something to loop)
do something with $someArray here;
?
is PHP using the same amount of memory as this:
?php
$someArray=array("this","that","those");
somefunction($someArray);
function somefunction($ref_to_array)
{
for(something to loop)
do something with $ref_to_array here;
}
?
because it seems to me, that the script gets a little slower when I
introduce the function method.
. Christian Dechery (lemming)
. http://www.tanamesa.com.br
. Gaita-L Owner / Web Developer
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
