RE: [PHP-DEV] CGI quick cleanup
At 09:33 AM 11/26/2001 +, Sam Liddicott wrote: -Original Message- From: Andi Gutmans [mailto:[EMAIL PROTECTED]] Sent: 24 November 2001 01:21 To: Sam Liddicott; Sam Liddicott; [EMAIL PROTECTED] Subject: RE: [PHP-DEV] CGI quick cleanup The problem you are experiencing is due to the fast cache. Edit Zend/zend_fast_cache.h and change: # define ZEND_ENABLE_FAST_CACHE 1 to: # define ZEND_ENABLE_FAST_CACHE 0 I'm not sure what fast cache is, is lack of this likely to slow down the processing of enormous hashed arrays? I think it really depends on the exact memory allocation pattern of your program. I have a feeling that in your case it'll be much faster without the fast cache but you can benchmark it. Make sure you do a complete rebuild. Tomorrow I'll try and think of what the best way to fix it is. (3:20 AM here :) To go to bed late is to be up early (paraphrased from Shakespeare summit-or-other) :) Andi -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
RE: [PHP-DEV] CGI quick cleanup
The problem you are experiencing is due to the fast cache.
Edit Zend/zend_fast_cache.h and change:
# define ZEND_ENABLE_FAST_CACHE 1
to:
# define ZEND_ENABLE_FAST_CACHE 0
Make sure you do a complete rebuild.
Tomorrow I'll try and think of what the best way to fix it is. (3:20 AM here :)
Andi
At 01:37 PM 11/23/2001 +, Sam Liddicott wrote:
Here's a sample script that does most of the sorts of stuff I was doing
apart from database work.
Note how long it takes to exit after finishing.
#! /usr/bin/php -q
?php
ini_Set(max_execution_time,0);
ini_Set(memory_limit,500M);
class thingy {
function thingy($c) {
if ($c0) $this-ref=new thingy($c-1);
}
}
$stash=array();
$max=50;
$start=time();
for($i=0;$i$max;$i++) {
$r=rand(0,300);
$stash[$r][]=new thingy(rand(0,10));
echo \rUse: .floor($i/$max*100).% ;
}
echo \n;
$mid=time();
$max=count($stash);
$c=0;
foreach(array_keys($stash) as $key) {
unset($stash[$key]);
$c++;
echo \rFree: .floor($c/$max*100).% ;
}
unset($stash);
echo \n;
$done=time();
print Use: .($mid-$start).\n;
print Free: .($done-$mid).\n;
?
--
PHP Development Mailing List http://www.php.net/
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
RE: [PHP-DEV] CGI quick cleanup
I should add that without the kill -5 it takes longer for the script to finally exit than it took to execute! Sam -Original Message- From: Sam Liddicott [mailto:[EMAIL PROTECTED]] Sent: 23 November 2001 10:35 To: [EMAIL PROTECTED] Subject: [PHP-DEV] CGI quick cleanup I am using PHP for a system script to import TV listings to a database. It works well but creates 20,000 to 30,000 hash elements with various relationships and it *seems* to take exponentially longer for the script to actually stop running after it reaches the exit statement the more items there are. I also note that even though max execution time is set to zero and the script runs for a long time, I still get a max-execution error at 30 seconds after the script has been trying to exit (although the script runs for many minutes). 30 seconds is the timeout in the php.ini which I override with ini_set. I think this long shutdown time is PHP's per-page cleanup freeing all the objects, which is not so important if running as a CGI. Currently I am quitting with posix_kill(posix_getpid(),5) as it is pretty quick but I wondered if the per-page cleanup which is only important when running as an apache module could be disabled in cgi mode. Sam -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED] -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
I have noticed the same problem with a scipt that used a very large array (~160 MB). The script run time was around 35 seconds, while it took over 4 minutes to shut down! Same amount of time was used in trying to unset() the array. Edin - Original Message - From: Sam Liddicott [EMAIL PROTECTED] To: Sam Liddicott [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Friday, November 23, 2001 11:41 AM Subject: RE: [PHP-DEV] CGI quick cleanup I should add that without the kill -5 it takes longer for the script to finally exit than it took to execute! Sam -Original Message- From: Sam Liddicott [mailto:[EMAIL PROTECTED]] Sent: 23 November 2001 10:35 To: [EMAIL PROTECTED] Subject: [PHP-DEV] CGI quick cleanup I am using PHP for a system script to import TV listings to a database. It works well but creates 20,000 to 30,000 hash elements with various relationships and it *seems* to take exponentially longer for the script to actually stop running after it reaches the exit statement the more items there are. I also note that even though max execution time is set to zero and the script runs for a long time, I still get a max-execution error at 30 seconds after the script has been trying to exit (although the script runs for many minutes). 30 seconds is the timeout in the php.ini which I override with ini_set. I think this long shutdown time is PHP's per-page cleanup freeing all the objects, which is not so important if running as a CGI. Currently I am quitting with posix_kill(posix_getpid(),5) as it is pretty quick but I wondered if the per-page cleanup which is only important when running as an apache module could be disabled in cgi mode. Sam -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
Hi Edin! On Fri, 23 Nov 2001, Edin Kadribasic wrote: I have noticed the same problem with a scipt that used a very large array (~160 MB). The script run time was around 35 seconds, while it took over 4 minutes to shut down! Same amount of time was used in trying to unset() the array. [wild guess] probably the memory deallocation is the one expensive here. BTW, don't use large arrays in PHP as Apache module cause when the process grows requesting more memory (brk(2)) it cannot release it back to the system (because the httpd process is still running). -- teodor -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
RE: [PHP-DEV] CGI quick cleanup
-Original Message- From: Teodor Cimpoesu [mailto:[EMAIL PROTECTED]] Sent: 23 November 2001 10:54 To: [EMAIL PROTECTED] Subject: Re: [PHP-DEV] CGI quick cleanup Hi Edin! On Fri, 23 Nov 2001, Edin Kadribasic wrote: I have noticed the same problem with a scipt that used a very large array (~160 MB). The script run time was around 35 seconds, while it took over 4 minutes to shut down! Same amount of time was used in trying to unset() the array. [wild guess] probably the memory deallocation is the one expensive here. BTW, don't use large arrays in PHP as Apache module cause when the process grows requesting more memory (brk(2)) it cannot release it back to the system (because the httpd process is still running). To solve THIS problem I added ap_child_terminate support to php so you could request apache to terminate after the page, though this seems to have been dropped in later releases. The patch was accepted and did appear in at least one release, I don't know why it went. But our problem is not as an apache module but as a system script, using php -q Sam -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
On Fri, Nov 23, 2001 at 11:50:17AM +0100, Edin Kadribasic wrote : I have noticed the same problem with a scipt that used a very large array (~160 MB). The script run time was around 35 seconds, while it took over 4 minutes to shut down! Same amount of time was used in trying to unset() the array. Hmm .. your script... is this something which other can easily reproduce with the same datainput or is it specific because of the database, the content, etc? If the latter, can you in more detail describe the how much data and the structure of the data put into the arary with takes up 160mb? Maybe we can track something down the available profilers. - Markus -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
I have noticed the same problem with a scipt that used a very large array (~160 MB). The script run time was around 35 seconds, while it took over 4 minutes to shut down! Same amount of time was used in trying to unset() the array. [wild guess] probably the memory deallocation is the one expensive here. Probably true. But there is a big performance problem here, and I think it should be looked at. I cannot belive that 4 minutes to deallocte 160 MB ram is the best we can do on a 1 GHz machine with 1GB ram. BTW, don't use large arrays in PHP as Apache module cause when the process grows requesting more memory (brk(2)) it cannot release it back to the system (because the httpd process is still running). I'm aware of this, but this was command line script so that was not a problem. Edin -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
On Fri, Nov 23, 2001 at 11:01:03AM -, Sam Liddicott wrote :
To solve THIS problem I added ap_child_terminate support to php so you could
request apache to terminate after the page, though this seems to have been
dropped in later releases.
The patch was accepted and did appear in at least one release, I don't know
why it went.
I found this in php4/sapi/apache/php_apache.c :
/* {{{ proto string child_terminate()
Get and set Apache request notes */
PHP_FUNCTION(apache_child_terminate)
{
#ifndef MULTITHREAD
if (AP(terminate_child)) {
ap_child_terminate( ((request_rec *)SG(server_context))
);
} else { /* tell them to get lost! */
php_error(E_WARNING, apache.child_terminate is
disabled);
}
#else
php_error(E_WARNING, apache_child_terminate() is not
supported in this build);
#endif
}
/* }}} */
However, it doesn't seem to be documented (couldn't find it
in the manual). But yes, not really relevant because you're
using a system script.
- Markus
--
PHP Development Mailing List http://www.php.net/
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
RE: [PHP-DEV] CGI quick cleanup
-Original Message- From: Markus Fischer [mailto:[EMAIL PROTECTED]] Sent: 23 November 2001 11:16 To: Sam Liddicott Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: [PHP-DEV] CGI quick cleanup On Fri, Nov 23, 2001 at 11:01:03AM -, Sam Liddicott wrote : To solve THIS problem I added ap_child_terminate support to php so you could request apache to terminate after the page, though this seems to have been dropped in later releases. The patch was accepted and did appear in at least one release, I don't know why it went. I found this in php4/sapi/apache/php_apache.c : [apache_child_terminate function deleted] heh! Thanks. However, it doesn't seem to be documented (couldn't find it in the manual). But yes, not really relevant because you're using a system script. Hmm. What do I need to do to get this documented then? Thanks Sam -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
On Fri, Nov 23, 2001 at 11:28:02AM -, Sam Liddicott wrote : Hmm. What do I need to do to get this documented then? Oh well, talk so some phpdoc guy directly or just open a bug report about it I guess? - MArkus -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
RE: [PHP-DEV] CGI quick cleanup
Here's a sample script that does most of the sorts of stuff I was doing
apart from database work.
Note how long it takes to exit after finishing.
#! /usr/bin/php -q
?php
ini_Set(max_execution_time,0);
ini_Set(memory_limit,500M);
class thingy {
function thingy($c) {
if ($c0) $this-ref=new thingy($c-1);
}
}
$stash=array();
$max=50;
$start=time();
for($i=0;$i$max;$i++) {
$r=rand(0,300);
$stash[$r][]=new thingy(rand(0,10));
echo \rUse: .floor($i/$max*100).% ;
}
echo \n;
$mid=time();
$max=count($stash);
$c=0;
foreach(array_keys($stash) as $key) {
unset($stash[$key]);
$c++;
echo \rFree: .floor($c/$max*100).% ;
}
unset($stash);
echo \n;
$done=time();
print Use: .($mid-$start).\n;
print Free: .($done-$mid).\n;
?
--
PHP Development Mailing List http://www.php.net/
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
Re: [PHP-DEV] CGI quick cleanup
The reason for this is most probably because during shutdown we decrement many reference counts and free each memory block separately. Theoretically we could just exit without doing any freeing of memory but we have to find a solution for freeing resources (maybe directly in the resource list and not via reference counting). If you can put together a short script which demonstrates the problem (without using external sources such as databases) then I will try and take a look at it and see if we need to improve the current code or go for a different approach. Also, I hope you're running in non-debug mode because debug slows down memory allocation/freeing significantly. Thanks, Andi At 10:35 AM 11/23/2001 +, Sam Liddicott wrote: I am using PHP for a system script to import TV listings to a database. It works well but creates 20,000 to 30,000 hash elements with various relationships and it *seems* to take exponentially longer for the script to actually stop running after it reaches the exit statement the more items there are. I also note that even though max execution time is set to zero and the script runs for a long time, I still get a max-execution error at 30 seconds after the script has been trying to exit (although the script runs for many minutes). 30 seconds is the timeout in the php.ini which I override with ini_set. I think this long shutdown time is PHP's per-page cleanup freeing all the objects, which is not so important if running as a CGI. Currently I am quitting with posix_kill(posix_getpid(),5) as it is pretty quick but I wondered if the per-page cleanup which is only important when running as an apache module could be disabled in cgi mode. Sam -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED] -- PHP Development Mailing List http://www.php.net/ To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
