Re: [PHP] PHP Memory Leak

[Jim Lucas]

Sascha Braun wrote:

Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
foreach ($array as $key => value) {
if ($value['element'] == 'test1') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
} elseif ($value['element'] == 'test2') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}


I would do a in_array here..

if (in_array($value['element'], array('test1', 'test2') ) ) {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}



if ($num_new > 0) {
where are you lowering this number?  From what I can tell, it is always 
going to be 4




// this part causes the memory leak

for ($i = 0; $i < $num_new; $i++) {


you have a $i - 0   (a minus sign) not equals


echo "sgasdgga";
}
}   
}
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,

Thank you!


np



I hope for a solution



How about that?

--
Jim Lucas


"Perseverance is not a long race;
it is many short races one after the other"

Walter Elliot



"Some men are born to greatness, some achieve greatness,
and some have greatness thrust upon them."

Twelfth Night, Act II, Scene V
by William Shakespeare

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP Memory Leak

[Casey]

On Dec 6, 2007, at 3:15 PM, Sascha Braun <[EMAIL PROTECTED]>  
wrote:



Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
   foreach ($array as $key => value)

Typo on above line?

{
   if ($value['element'] == 'test1') {
   foreach ($value['data'] as $skey => $svalue) {
   echo $svalue;
   }
   } elseif ($value['element'] == 'test2') {
   foreach ($value['data'] as $skey => $svalue) {
   echo $svalue;
   }
   }
   if ($num_new > 0) {

   // this part causes the memory leak

   for ($i = 0; $i < $num_new; $i++) {
   echo "sgasdgga";
   }
   }
   }
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,

Thank you!

I hope for a solution

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP Memory Leak

[Chris]

Sascha Braun wrote:

Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
foreach ($array as $key => value) {
if ($value['element'] == 'test1') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
} elseif ($value['element'] == 'test2') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}
if ($num_new > 0) {

// this part causes the memory leak

for ($i = 0; $i < $num_new; $i++) {
echo "sgasdgga";
}
}   
}
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,


If you don't know how are we supposed to know? :)

Add

memory_get_usage() calls all over the place and see what's going on.

eg:

error_log(__LINE__ . "\t" . memory_get_usage() . "\n", 3, 
'/path/to/log.file');


at various spots and go from there.

--
Postgresql & php tutorials
http://www.designmagick.com/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] PHP Memory Leak

[Sascha Braun]

Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
foreach ($array as $key => value) {
if ($value['element'] == 'test1') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
} elseif ($value['element'] == 'test2') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}
if ($num_new > 0) {

// this part causes the memory leak

for ($i = 0; $i < $num_new; $i++) {
echo "sgasdgga";
}
}   
}
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,

Thank you!

I hope for a solution

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory leak - how to find it?

[Richard Lynch]

On Mon, July 31, 2006 6:23 am, Robin Getz wrote:
> I am trying to debug a php script that I downloaded, which has a
> memory
> leak in it.
>
> I was looking for a way to find what variables were in php's memory,
> and
> what size, they were, but I couldn't find anything?
>
> The script is a off-line wiki conversion tool (walks through a wiki to
> create a bunch of html files for off line viewing). As the tools walks
> the
> files, and does the conversion, I can see the memory consumption go up
> and
> up as it walks the files, until it hits the mem limit, and crashes.
>
> Any suggestions appreciated.

When you run the tool, just give the command-line options to PHP to
give it more RAM for memory_limit. :-)

Finding the memory leak in a Wiki-walker could take forever.

-- 
Like Music?
http://l-i-e.com/artists.htm

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory leak - how to find it?

[chris smith]

On 7/31/06, Robin Getz <[EMAIL PROTECTED]> wrote:

I am trying to debug a php script that I downloaded, which has a memory
leak in it.

I was looking for a way to find what variables were in php's memory, and
what size, they were, but I couldn't find anything?

The script is a off-line wiki conversion tool (walks through a wiki to
create a bunch of html files for off line viewing). As the tools walks the
files, and does the conversion, I can see the memory consumption go up and
up as it walks the files, until it hits the mem limit, and crashes.


Depending on how "big" the wiki is, if it's creating references to
other files it will need a lot of memory to remember all the different
links it has to create (phpdocumentor has a similar issue/reason).

--
Postgresql & php tutorials
http://www.designmagick.com/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory leak - how to find it?

[David Tulloh]

Robin Getz wrote:
> I am trying to debug a php script that I downloaded, which has a memory
> leak in it.
> 
> I was looking for a way to find what variables were in php's memory, and
> what size, they were, but I couldn't find anything?
> 
> The script is a off-line wiki conversion tool (walks through a wiki to
> create a bunch of html files for off line viewing). As the tools walks
> the files, and does the conversion, I can see the memory consumption go
> up and up as it walks the files, until it hits the mem limit, and crashes.
> 
> Any suggestions appreciated.
> 
> Thanks
> -Robin
> 

xdebug has some advanced tools to help you track down these kind of
problems.  I believe that there are also some other similar php
debugging extensions.


David

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] memory leak - how to find it?

[Robin Getz]

I am trying to debug a php script that I downloaded, which has a memory 
leak in it.


I was looking for a way to find what variables were in php's memory, and 
what size, they were, but I couldn't find anything?


The script is a off-line wiki conversion tool (walks through a wiki to 
create a bunch of html files for off line viewing). As the tools walks the 
files, and does the conversion, I can see the memory consumption go up and 
up as it walks the files, until it hits the mem limit, and crashes.


Any suggestions appreciated.

Thanks
-Robin

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory leak when referencing associative arrays

[Gonzalo MC]

Gonzalo MC escribió:


Jochem Maas escribió:


Gonzalo MC wrote:


Hi Jochem,

Thank you very much for your reply! 




no problem :-)

Your reply lend me this morning to had a look again at the code, and 
I have solved more than half of the trouble caused by a leak on a 
PhpGtk function! :-)


Sorry for the more long reply than it should be, but I hope you'll 
have the time to read it. Unfortunately,  I have not fount an 
easyier way to explain the thing and get to something with meaning.




well I read it, but I didn't understand all of it. not to worry.

as a sidenote - reading about the objects (some gtk style wotsit) in 
your codebase
I got the impression that the unfreed memory is possibly [mostly] to 
do with those -
my memory usage problems are always to do with objects (remember in 
php that
object variables basically work semantically like scalar values as 
opposed to php5
where object variables are handles to objectsa and effectively[, 
always,] work like

[php] references of normal variables)

I'd concentrate on getting those objects 'tuned' - generous but 
highly controlled
use of the '&' symbol to force objects to be passed by reference 
where possible/relevant

(e.g. when using the 'new MyObject;' syntax) can also help.

...


Hi again,

Yeah you're right the memory usage problems always have to do with 
objects on  php4, and using '&' is of great help, but in my case, with 
the actual implementation, I've tested almost all the ways in the 
exact points where data is handled, and fount no reliable way for 
reusing reserved memory from the interpreter other than going to 
functional style programming and avoid setting object properties, etc.


Fortunately, my application almost does not create any object other 
than in the beginning/initialization of and for the GUI, for what I 
have some helper classes to abstract interfaces, but they are all 
instantiated on the program startup and the only objects I do create 
"on the run time" are perhaps the new style copyes according to each 
new insert or update to the existing widgets on screen, and they are 
almost all referenced byref using & when adding pixmaps and such - 
note the $astyle = gtk::rc_get_style($ctree); line per example is only 
done once before all items loop and it does not take any memory, not 
the case when creating or transfering a widget from a way to another 
where the & really matters- but the trouble is when you call for each 
node in the loop to the $astyle->copy(); method per example-.


All the data I generate, update or read are arrays with numeric or 
alfanumeric keys, in wich I place another array with some alfanumeric 
keys.


The data is generated by an object. It is stored in global variables, 
and some references to these are keept on that or other object 
properties, when processing or updating data based on the last data 
received.  I think the trick was to put these references to the global 
array items, outside these objects, in the global namespace too, but 
only worked when the data was processed from plain functions, 
-actually the processing work is done from a class instance that 
extends another root class-.


Maybe it haves something to do with the processing and gui update 
loops, fired by gtk_timeout's from inside objects calling object 
methods... Maybe on refactoring, now I have more knowledge about whats 
going on, I will notice of.


I expect to get soon the time needed to refactor again this and I'll 
post the concrete workaround for my case, perhaps could help somebody 
to get a more clear idea of ways to avoid these memory usage issues on 
php4 / PhpGtk1. Until then, I'll though a good plan so when 
refactoring going in stages of  "migration" so I could really find the 
real nature of the issue -object properties are references to some 
globals / processing of globals fired by gtk_timeouts calling object 
methods by reference / whatever it could be-.


And thank you again for your response. I know perhaps it does not 
matter much but... :-) thats it.


Best Regards,
Gonzalo.

Sorry but I forget to say that when loading data from a file, -so a big 
process is done without updating the GUI where is the only place objects 
are perhaps more relevant- the program gets a lots of megs of ram, only 
with the processing of the global arrays from within an object...  So it 
has to do with the processing of that data from an object.


For a moment I though the data I'd willing to reuse is maybe leak in 
some place... not in the process itself, but on the other way I do 
remember clearly how I did get improved this part of the process 
-loading from file, processing, and finally do a GUI update with 
results, and do it again with another file, and have only a small amount 
extra of memory used- while doing it from an object lends to a lot of 
memory not being reused -or being really leak-.


Regards,
Gonzalo

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub

Re: [PHP] memory leak when referencing associative arrays

[Gonzalo MC]

Jochem Maas escribió:


Gonzalo MC wrote:


Hi Jochem,

Thank you very much for your reply! 



no problem :-)

Your reply lend me this morning to had a look again at the code, and 
I have solved more than half of the trouble caused by a leak on a 
PhpGtk function! :-)


Sorry for the more long reply than it should be, but I hope you'll 
have the time to read it. Unfortunately,  I have not fount an easyier 
way to explain the thing and get to something with meaning.



well I read it, but I didn't understand all of it. not to worry.

as a sidenote - reading about the objects (some gtk style wotsit) in 
your codebase
I got the impression that the unfreed memory is possibly [mostly] to 
do with those -
my memory usage problems are always to do with objects (remember in 
php that
object variables basically work semantically like scalar values as 
opposed to php5
where object variables are handles to objectsa and effectively[, 
always,] work like

[php] references of normal variables)

I'd concentrate on getting those objects 'tuned' - generous but highly 
controlled
use of the '&' symbol to force objects to be passed by reference where 
possible/relevant

(e.g. when using the 'new MyObject;' syntax) can also help.

...


Hi again,

Yeah you're right the memory usage problems always have to do with 
objects on  php4, and using '&' is of great help, but in my case, with 
the actual implementation, I've tested almost all the ways in the exact 
points where data is handled, and fount no reliable way for reusing 
reserved memory from the interpreter other than going to functional 
style programming and avoid setting object properties, etc.


Fortunately, my application almost does not create any object other than 
in the beginning/initialization of and for the GUI, for what I have some 
helper classes to abstract interfaces, but they are all instantiated on 
the program startup and the only objects I do create "on the run time" 
are perhaps the new style copyes according to each new insert or update 
to the existing widgets on screen, and they are almost all referenced 
byref using & when adding pixmaps and such - note the $astyle = 
gtk::rc_get_style($ctree); line per example is only done once before all 
items loop and it does not take any memory, not the case when creating 
or transfering a widget from a way to another where the & really 
matters- but the trouble is when you call for each node in the loop to 
the $astyle->copy(); method per example-.


All the data I generate, update or read are arrays with numeric or 
alfanumeric keys, in wich I place another array with some alfanumeric keys.


The data is generated by an object. It is stored in global variables, 
and some references to these are keept on that or other object 
properties, when processing or updating data based on the last data 
received.  I think the trick was to put these references to the global 
array items, outside these objects, in the global namespace too, but 
only worked when the data was processed from plain functions, -actually 
the processing work is done from a class instance that extends another 
root class-.


Maybe it haves something to do with the processing and gui update loops, 
fired by gtk_timeout's from inside objects calling object methods... 
Maybe on refactoring, now I have more knowledge about whats going on, I 
will notice of.


I expect to get soon the time needed to refactor again this and I'll 
post the concrete workaround for my case, perhaps could help somebody to 
get a more clear idea of ways to avoid these memory usage issues on php4 
/ PhpGtk1. Until then, I'll though a good plan so when refactoring going 
in stages of  "migration" so I could really find the real nature of the 
issue -object properties are references to some globals / processing of 
globals fired by gtk_timeouts calling object methods by reference / 
whatever it could be-.


And thank you again for your response. I know perhaps it does not matter 
much but... :-) thats it.


Best Regards,
Gonzalo.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory leak when referencing associative arrays

[Jochem Maas]

Gonzalo MC wrote:

Hi Jochem,

Thank you very much for your reply! 


no problem :-)

Your reply lend me this morning to had a look again at the code, and I 
have solved more than half of the trouble caused by a leak on a PhpGtk 
function! :-)


Sorry for the more long reply than it should be, but I hope you'll have 
the time to read it. Unfortunately,  I have not fount an easyier way to 
explain the thing and get to something with meaning.


well I read it, but I didn't understand all of it. not to worry.

as a sidenote - reading about the objects (some gtk style wotsit) in your 
codebase
I got the impression that the unfreed memory is possibly [mostly] to do with 
those -
my memory usage problems are always to do with objects (remember in php that
object variables basically work semantically like scalar values as opposed to 
php5
where object variables are handles to objectsa and effectively[, always,] work 
like
[php] references of normal variables)

I'd concentrate on getting those objects 'tuned' - generous but highly 
controlled
use of the '&' symbol to force objects to be passed by reference where 
possible/relevant
(e.g. when using the 'new MyObject;' syntax) can also help.

...

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory leak when referencing associative arrays

[Gonzalo MC]

Hi Jochem,

Thank you very much for your reply!  

Your reply lend me this morning to had a look again at the code, and I 
have solved more than half of the trouble caused by a leak on a PhpGtk 
function! :-)


Sorry for the more long reply than it should be, but I hope you'll have 
the time to read it. Unfortunately,  I have not fount an easyier way to 
explain the thing and get to something with meaning.


Perhaps the more comprimising memory leak happened when my application 
is "active", receiving and processing the data received from one or 
several devices in real-time -well, it does a 300ms. delay until the 
main next loop iteration starts-, and this trouble was limiting a lot 
the amount of time the app. could be running before the OS have to start 
swapping pages to disk, caused by the memory being leaked on each 
iteration, -of course, it depends on the size or number of the 
"incoming" data per minute, etc. etc.-, in wich situation I have to 
finish the app. and restart it again for "clean" state, but in a nomal 
case use test, it was consuming about a megabite each minute or two, 
when each "round" is completed and data is received and processed from 
all devices.


The improvement it is as simpler as not using styles for every node 
insert as some GtkCtrees are populated when receiving and processing 
data) and now it "only" takes about 300kb when before was taking one 
meg.  It is a only a PHP-Gtk issue as the leak is produced when copying 
a widget style, a must if you want to change the style of the nodes 
being inserted-updated, and not doing this helped a lot (I was doing 
this about 200 times (one time per node) in minute or two wich lead to a 
high amount of mem leaked...)


I really did not expect to find such a leak on a simple "$style = 
$astyle->copy()" where "$astyle = gtk::rc_get_style($ctree);" !!-.


Anyway a problem persist, as 300kb per minute or two are a lot of 
kilobytes if you though in that time I "only" add about 100-125 items 
and update another 50 (each item have from 50 to 200 bytes of data- to a 
pair of global associative arrays.  And I really don't know exactly 
where it is leaking, I don't know if it leaks on every iteration, or 
maybe the amount of memory "used" while the application runs is what it 
should, but then later when you stop the actual processing, save the 
data, and want to start a new run -all variables are unset-, the memory 
used stays there... and "rewriting" to the same global variables with 
-probably- the same keys -but with different data- when receiving data 
and processing from the devices again, does not reuse the previouly 
reserved memory... it continues eating more and more. Or if you load 
previously saved data from disk -it only saves the "incoming data" and 
all items are re-calculated always- the same happens, more and more 
memory used and nothing is reused or freed.   And if I do some runs, or 
load some data files, the application gets so many megs. and I need to 
restart it.


I have no circular references on the arrays, and not a lot of references 
at all. And removing some caused the app. to get more memory than 
before, as I experienced some months ago. Only some associative arrays 
being referenced with or just generated and processed from within three 
objects.


I though I could ask for some help regards the garbage collector, so I 
see better what is happening. I haven't got yet the time to look at the 
sources and try to understand how does it work and that... :-) it could 
be really a pain. And I can't update to a newer php version... as I 
think it could help.


But on the other way, the actual trouble was "almost" solved some months 
ago when I did some tests -very succesful I'll say regards the "not 
reused" memory between application runs, so the memory used in a "run" 
was used again for the new data the next time the receiving/processing 
started, or when loading saved data-, but as it didn't help much in the 
memory used while processing, that not helped much for the main trouble, 
and felt lost 'cause though it was likely the same issue and could not 
improve more the code.


I achieved that by refactoring a big piece of code and doing all the 
"main" processing of global arrays from some plain functions, outside 
the objects where it was/is done, but sadly that lend to some bugs -i 
did it in hurry- and I had not the time to workaround them, and have to 
get back to the previous version to continue developement with more 
"stable" code.


Now that I have found where the most memory was being leaked when the 
application is active receiving data -that was really the more 
compromising thing- I'll have to find the time to refactor carefully the 
main processing loop to plain functions -a 170kb. code class- and not 
get caught by bugs... again :-)


Just a matter of processing a lot of global data within objects ?

Best Regards,
Gonzalo.


Jochem Maas escribió:


Gonzalo MC wrote:


Hi all,

I'm having some pain 

Re: [PHP] memory leak when referencing associative arrays

[Gonzalo MC]

Hi Jochem,

Thank you very much for your reply!  

Your reply lend me this morning to had a look again at the code, and I 
have solved more than half of the trouble caused by a leak on a PhpGtk 
function! :-)


Sorry for the more long reply than it should be, but I hope you'll have 
the time to read it. Unfortunately,  I have not fount an easyier way to 
explain the thing and get to something with meaning.


Perhaps the more comprimising memory leak happened when my application 
is "active", receiving and processing the data received from one or 
several devices in real-time -well, it does a 300ms. delay until the 
main next loop iteration starts-, and this trouble was limiting a lot 
the amount of time the app. could be running before the OS have to start 
swapping pages to disk, caused by the memory being leaked on each 
iteration, -of course, it depends on the size or number of the 
"incoming" data per minute, etc. etc.-, in wich situation I have to 
finish the app. and restart it again for "clean" state, but in a nomal 
case use test, it was consuming about a megabite each minute or two, 
when each "round" is completed and data is received and processed from 
all devices.


The improvement it is as simpler as not using styles for every node 
insert as some GtkCtrees are populated when receiving and processing 
data) and now it "only" takes about 300kb when before was taking one 
meg.  It is a only a PHP-Gtk issue as the leak is produced when copying 
a widget style, a must if you want to change the style of the nodes 
being inserted-updated, and not doing this helped a lot (I was doing 
this about 200 times (one time per node) in minute or two wich lead to a 
high amount of mem leaked...)


I really did not expect to find such a leak on a simple "$style = 
$astyle->copy()" where "$astyle = gtk::rc_get_style($ctree);" !!-.


Anyway a problem persist, as 300kb per minute or two are a lot of 
kilobytes if you though in that time I "only" add about 100-125 items 
and update another 50 (each item have from 50 to 200 bytes of data- to a 
pair of global associative arrays.  And I really don't know exactly 
where it is leaking, I don't know if it leaks on every iteration, or 
maybe the amount of memory "used" while the application runs is what it 
should, but then later when you stop the actual processing, save the 
data, and want to start a new run -all variables are unset-, the memory 
used stays there... and "rewriting" to the same global variables with 
-probably- the same keys -but with different data- when receiving data 
and processing from the devices again, does not reuse the previouly 
reserved memory... it continues eating more and more. Or if you load 
previously saved data from disk -it only saves the "incoming data" and 
all items are re-calculated always- the same happens, more and more 
memory used and nothing is reused or freed.   And if I do some runs, or 
load some data files, the application gets so many megs. and I need to 
restart it.


I have no circular references on the arrays, and not a lot of references 
at all. And removing some caused the app. to get more memory than 
before, as I experienced some months ago. Only some associative arrays 
being referenced with or just generated and processed from within three 
objects.


I though I could ask for some help regards the garbage collector, so I 
see better what is happening. I haven't got yet the time to look at the 
sources and try to understand how does it work and that... :-) it could 
be really a pain. And I can't update to a newer php version... as I 
think it could help.


But on the other way, the actual trouble was "almost" solved some months 
ago when I did some tests -very succesful I'll say regards the "not 
reused" memory between application runs, so the memory used in a "run" 
was used again for the new data the next time the receiving/processing 
started, or when loading saved data-, but as it didn't help much in the 
memory used while processing, that not helped much for the main trouble, 
and felt lost 'cause though it was likely the same issue and could not 
improve more the code.


I achieved that by refactoring a big piece of code and doing all the 
"main" processing of global arrays from some plain functions, outside 
the objects where it was/is done, but sadly that lend to some bugs -i 
did it in hurry- and I had not the time to workaround them, and have to 
get back to the previous version to continue developement with more 
"stable" code.


Now that I have found where the most memory was being leaked when the 
application is active receiving data -that was really the more 
compromising thing- I'll have to find the time to refactor carefully the 
main processing loop to plain functions -a 170kb. code class- and not 
get caught by bugs... again :-)


Just a matter of processing a lot of global data within objects ?

Best Regards,
Gonzalo.


Jochem Maas escribió:


Gonzalo MC wrote:


Hi all,

I'm having some pain 

Re: [PHP] memory leak when referencing associative arrays

[Jochem Maas]

Gonzalo MC wrote:

Hi all,

I'm having some pain with this process. I'll try to explain it well, my 
english is little bad... :-)


...

Does the php garbage collector keep track of uncollectable objects / 
zvals as python gc does?


It is some way to freed some memory I know it should be really 
unreferenced?  Some thing like a manual force to the garbage collection 
of a zval -associative array at php level-?


Perhaps should I use a different mechanism for storing the associative 
arrays so I could not suffer this issue?


I don't think there is anything you can do to force garbage collection.
note that the garbage collector is not capable of freeing memory for zvals
that contain circular references (IIRC) - e.g. 2 objects with properties
pointing to each other (I assume the same problem occurs if you have 2 arrays
with items that are references to each other [or each others items?]).

if you have circular references I would try to find a way to do it without
them - it should help memory consumption (assuming you even have any
references in your arrays)

I take you are unset()ing the relevant variables already in your code?



If my trouble is not very clear, please don't hesitate to ask me.

Thanks in advance.

Regards,
Gonzalo.



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] memory leak when referencing associative arrays

[Gonzalo MC]

Hi all,

I'm having some pain with this process. I'll try to explain it well, my 
english is little bad... :-)


I've got two main loops, handled by timeouts.  One loop calls to an 
object method every half second to do a computation. It reads from 
external device, creates helper arrays for calculations, and then 
creates/updates another array where the final calculations are stored.  
In the other loop, fired by events or some circunstances, I loop thru 
the calculations array and get the new state for some values. This is 
not done on the same object.


I'm having some memory leak trouble while doing that from within objects 
using global variables to store the associative arraya, so it seems to 
me that some references are keep by php to these data I do generate and 
destroy or update on each computation loop, and that memory it is not 
freed anytime until program execution ends. 

If I do the same computation, but I don't do the data lookups to the 
associative arrays, the memory is normaly fred as expected -I noticed 
the same trouble with this that depends on how the arrays are handled or 
accesed before deleting them, when trying to test for that problem-. But 
if I do the array lookups (i mean with this to do a loop to the array to 
get all item updates ...) to data is being processed / updated from 
other objects, the trouble I explained begin...


It is somewhat complex to give an example to reproduce the whole 
thing...   Notice I use php4.4.1, and sadly, it is a long running script 
as it handles some phpgtk interface.


I noticed the same results when converting these global variables to 
local object properties, sadly. It is possible on PHP4 to call on 
runtime to the garbage collector? possibly via a php extension / C code?


Does the php garbage collector keep track of uncollectable objects / 
zvals as python gc does?


It is some way to freed some memory I know it should be really 
unreferenced?  Some thing like a manual force to the garbage collection 
of a zval -associative array at php level-?


Perhaps should I use a different mechanism for storing the associative 
arrays so I could not suffer this issue?


If my trouble is not very clear, please don't hesitate to ask me.

Thanks in advance.

Regards,
Gonzalo.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[cron]

I think the php GC only kicks in at the end of a script some calls to 
mysql_free_result might help




Angelo

- Original Message - 
From: "Richard Lynch" <[EMAIL PROTECTED]>

To: <[EMAIL PROTECTED]>
Cc: 
Sent: Saturday, October 22, 2005 11:40 PM
Subject: Re: [PHP] Memory Leak?



On Sat, October 22, 2005 9:24 pm, [EMAIL PROTECTED] wrote:

Sucks that you don't have more control over the incoming data.


Just for the record, for when the source of this data reads this thread.

I am ECSTATIC to have this data, period.

:-)

I know we'll work out a viable solution in time, even if replication
is not in the cards.

--
Like Music?
http://l-i-e.com/artists.htm

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php




--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[Richard Lynch]

On Sat, October 22, 2005 9:24 pm, [EMAIL PROTECTED] wrote:
> Sucks that you don't have more control over the incoming data.

Just for the record, for when the source of this data reads this thread.

I am ECSTATIC to have this data, period.

:-)

I know we'll work out a viable solution in time, even if replication
is not in the cards.

-- 
Like Music?
http://l-i-e.com/artists.htm

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[tg-php]

One other thing to keep in mind.  Connecting to a database and disconnecting 
takes some amount of time.  If you do it in batches of 10, 100, whatever then 
make sure you only connect and disconnect once or else you won't have any time 
savings at all (and will burden the server unnecessarily).

Sucks that you don't have more control over the incoming data.

Good luck in finding a solution.

-TG

= = = Original message = = =

On Sat, October 22, 2005 5:04 pm, [EMAIL PROTECTED] wrote:
> Does it make a difference if instead of one record at a time, you pull
> 10, 20, 100...   might speed up your process a little and if it
> doesn't use any more memory, then why not?

I could make a buffer of N lines and read them, I guess.

I kinda figured at 1 line a second, PHP and the OS would have no
problem buffering from the disk.

> Another thought..  can you use database replication here?  So instead
> of dumping all the records and processing and inserting the ones that
> are new, you'd only be acting on what's changed since the last backup.
>  With real replication, you can do this realtime (or at least
> semi-realtime), but you could simulate replication in your PHP script
> as well.

This data's coming from an external source.

Replication is probably not gonna happen.

Though I am going to ask for a timestamp column so I can ignore rows
that haven't changed since my last import.

That should kill about 90% of the MySQL processing, which seems to be
the biggest CPU sink here.

If that happens, I can probably change the sleep to usleep and a
quarter second or so.


___
Sent by ePrompter, the premier email notification software.
Free download at http://www.ePrompter.com.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[Richard Lynch]

On Sat, October 22, 2005 5:04 pm, [EMAIL PROTECTED] wrote:
> Does it make a difference if instead of one record at a time, you pull
> 10, 20, 100...   might speed up your process a little and if it
> doesn't use any more memory, then why not?

I could make a buffer of N lines and read them, I guess.

I kinda figured at 1 line a second, PHP and the OS would have no
problem buffering from the disk.

> Another thought..  can you use database replication here?  So instead
> of dumping all the records and processing and inserting the ones that
> are new, you'd only be acting on what's changed since the last backup.
>  With real replication, you can do this realtime (or at least
> semi-realtime), but you could simulate replication in your PHP script
> as well.

This data's coming from an external source.

Replication is probably not gonna happen.

Though I am going to ask for a timestamp column so I can ignore rows
that haven't changed since my last import.

That should kill about 90% of the MySQL processing, which seems to be
the biggest CPU sink here.

If that happens, I can probably change the sleep to usleep and a
quarter second or so.

-- 
Like Music?
http://l-i-e.com/artists.htm

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[Jasper Bryant-Greene]

On Sat, 2005-10-22 at 16:36 -0500, Richard Lynch wrote:
> I've written a script to munge and import 108,000+ records.
[snip]
> http://l-i-e.com/feedbaby/memory_leak.htm

It looks fine to me, but you might like to try accumulating the records
say up to 100 at a time and then doing extended INSERTs (if your MySQL
version supports them) like this:

INSERT INTO tablename
(col1, col2)
VALUES
('value1', 'value2'),
('value3', 'value4'),
('value5', 'value6'),
('value7', 'value7')

... and so on.

These are apparently much nicer for MySQL to process, which may result
in lower memory usage and faster operation.

-- 
Jasper Bryant-Greene
General Manager
Album Limited

e: [EMAIL PROTECTED]
w: http://www.album.co.nz/
p: 0800 4 ALBUM (0800 425 286) or +64 21 232 3303
a: PO Box 579, Christchurch 8015, New Zealand

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[Robert Cummings]

On Sat, 2005-10-22 at 17:36, Richard Lynch wrote:
> I've written a script to munge and import 108,000+ records.
> 
> To avoid spiking the server, I'm sleep()ing 1 second for each record.
> 
> So it takes 30+ hours to run, so what?
> 
> This data changes "daily" but not really much more often than that,
> mostly.
> 
> Anyway, it seems to be using an awful lot of RAM for what it's doing.
> 
> Like, 80 Meg or so.
> 
> php.ini memory_limit is set to 100M by my ISP.  I can use my own
> php.ini and change that, if needed.
> 
> Most of the fields are short, and the longest is maybe a varchar(255)
> and there are only ~15 fields.
> 
> There's only a couple $query strings in each iteration, a couple MySQL
> result handles, and 2 copies (raw and munged) of each field's data.
> 
> That don't sound like 80 Meg worth of data to this naive user.
> 
> I got worried about the RAM, so stopped the process and added some RAM
> usage calls, started it over, and am logging the RAM usage at each
> record.
> 
> I've written a "pretty" PNG graph and I'd like some experts to look at
> the graph, look at the code, and then tell me.
> 
> 1. Do I have a memory leak that is gonna kill me and I have to fix it?
> 
> 2. Is PHP's garbage-collection so non-aggressive that this is just
> "normal"?
> 
> 3. Is there some kind of 80/20 or 90/10 "rule" in the guts of PHP
> garbage-collection, so that reducing my memory_limit would just "fix"
> this?
> 
> 4. Can you spot any obvious/easy ways to alter the source to reduce
> memory usage without micro-managing or adding needless complications
> nor, perhaps most important, adding too much time onto the 30-hour
> process.
> 
> Below is a link to the graph, some commentary, and there's a link to
> the PHP source code at the bottom-right of the web page.
> 
> Hope all this isn't too presumptious...

The code looks fine, nothing being accumulated in the script itself so
my guess is that mysql is hanging onto something... possibly query
caching. Either that or as you say PHP itself though I can't say I've
experienced any huge memory issues in the past with long running
scripts. I'd definitely guess caching of some sort, especially since the
memory consumption levels out :)

Cheers,
Rob.
-- 
..
| InterJinn Application Framework - http://www.interjinn.com |
::
| An application and templating framework for PHP. Boasting  |
| a powerful, scalable system for accessing system services  |
| such as forms, properties, sessions, and caches. InterJinn |
| also provides an extremely flexible architecture for   |
| creating re-usable components quickly and easily.  |
`'

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory Leak?

[tg-php]

Because of the way it spikes and maintains, I'm guess that this is normal 
behavior, but we'll see what the real experts have to say.

In theory, if you allow a process 100MB of memory and it thinks it might need 
some stuff again later, it's likely to cache for faster reuse of that data 
later.

But putting all that aside, because I really have no idea, only hunches, I'm 
wondering if you might want to rethink how you're doing this in general.

Does it make a difference if instead of one record at a time, you pull 10, 20, 
100...   might speed up your process a little and if it doesn't use any more 
memory, then why not?

Another thought..  can you use database replication here?  So instead of 
dumping all the records and processing and inserting the ones that are new, 
you'd only be acting on what's changed since the last backup.  With real 
replication, you can do this realtime (or at least semi-realtime), but you 
could simulate replication in your PHP script as well.

The idea is to see what INSERT, UPDATE, DELETE and ALTER (maybe some other) 
commands are run on SourceDB and perform the same actions on DestinationDB.  
This can be done by either recording the actions for "playback" on the DestDB 
later or by performing the same actions on the DestDB at the same time as the 
SourceDB effectively mirroring the databases.

I'm not sure what 'data munging' is going on or if this is the right solution 
for you, but it might be worth checking into.

Here's the manual page for MySQL's replication:
http://dev.mysql.com/doc/refman/5.0/en/replication.html

But if you get the theory, you could easily keep a list of the 
INSERT/DELETE/UPDATE/etc that your scripts do on SourceDB and either do the 
same thing on DestDB right then, or "play it back" later like during nightly 
maintenance.

Good luck Richard!

-TG

= = = Original message = = =

I've written a script to munge and import 108,000+ records.

To avoid spiking the server, I'm sleep()ing 1 second for each record.

So it takes 30+ hours to run, so what?

This data changes "daily" but not really much more often than that,
mostly.

Anyway, it seems to be using an awful lot of RAM for what it's doing.

Like, 80 Meg or so.

php.ini memory_limit is set to 100M by my ISP.  I can use my own
php.ini and change that, if needed.

Most of the fields are short, and the longest is maybe a varchar(255)
and there are only ~15 fields.

There's only a couple $query strings in each iteration, a couple MySQL
result handles, and 2 copies (raw and munged) of each field's data.

That don't sound like 80 Meg worth of data to this naive user.

I got worried about the RAM, so stopped the process and added some RAM
usage calls, started it over, and am logging the RAM usage at each
record.

I've written a "pretty" PNG graph and I'd like some experts to look at
the graph, look at the code, and then tell me.

1. Do I have a memory leak that is gonna kill me and I have to fix it?

2. Is PHP's garbage-collection so non-aggressive that this is just
"normal"?

3. Is there some kind of 80/20 or 90/10 "rule" in the guts of PHP
garbage-collection, so that reducing my memory_limit would just "fix"
this?

4. Can you spot any obvious/easy ways to alter the source to reduce
memory usage without micro-managing or adding needless complications
nor, perhaps most important, adding too much time onto the 30-hour
process.

Below is a link to the graph, some commentary, and there's a link to
the PHP source code at the bottom-right of the web page.

Hope all this isn't too presumptious...

TIA!


___
Sent by ePrompter, the premier email notification software.
Free download at http://www.ePrompter.com.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory Leak?

[Richard Lynch]

I've written a script to munge and import 108,000+ records.

To avoid spiking the server, I'm sleep()ing 1 second for each record.

So it takes 30+ hours to run, so what?

This data changes "daily" but not really much more often than that,
mostly.

Anyway, it seems to be using an awful lot of RAM for what it's doing.

Like, 80 Meg or so.

php.ini memory_limit is set to 100M by my ISP.  I can use my own
php.ini and change that, if needed.

Most of the fields are short, and the longest is maybe a varchar(255)
and there are only ~15 fields.

There's only a couple $query strings in each iteration, a couple MySQL
result handles, and 2 copies (raw and munged) of each field's data.

That don't sound like 80 Meg worth of data to this naive user.

I got worried about the RAM, so stopped the process and added some RAM
usage calls, started it over, and am logging the RAM usage at each
record.

I've written a "pretty" PNG graph and I'd like some experts to look at
the graph, look at the code, and then tell me.

1. Do I have a memory leak that is gonna kill me and I have to fix it?

2. Is PHP's garbage-collection so non-aggressive that this is just
"normal"?

3. Is there some kind of 80/20 or 90/10 "rule" in the guts of PHP
garbage-collection, so that reducing my memory_limit would just "fix"
this?

4. Can you spot any obvious/easy ways to alter the source to reduce
memory usage without micro-managing or adding needless complications
nor, perhaps most important, adding too much time onto the 30-hour
process.

Below is a link to the graph, some commentary, and there's a link to
the PHP source code at the bottom-right of the web page.

Hope all this isn't too presumptious...

TIA!

http://l-i-e.com/feedbaby/memory_leak.htm

-- 
Like Music?
http://l-i-e.com/artists.htm

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory leak in PHP 5 / ODBC?

[chris . neale]

I'm having memory problems running a script in PHP 5 Beta on WINNT SP6a.

I am trying to generate 600 static HTML pages each containing a table of
data. I've taken out all the table formatting routines out the code below,
but it still replicates the same fault. The loop appends the SQL string with
the loop value and then calls ODBC_DO.

When I run it, PHP.exe shows up in task manager, eating away at memory fast
- about 1 meg every second gets consumed and it generates about 7000 page
faults every second too. [I've no idea what page faults are but there's lots
of them...]. But it still works, and the information comes out as expected.
The memory is then freed when PHP exits.

Here's the code.

$dbCon = odbc_connect('Test2', '', '');

for ($nval=1;$nval<=700;$nval++)
{
$query = "SELECT * from [tablename] where ID = ".$nval;
odbc_do($dbCon, $query);
}

odbc_close($dbCon); 

I've run this code not only from a web page, but also from the command line
with some additional code which waits for a user keystroke before continuing
to the next loop iteration, to slow down the process. But the same problem
occurred, so I think I've ruled out some kind of bottleneck. 

I expected the memory to remain fairly constant throughout execution, but it
keeps on rising and not always at a constant rate.

Can anyone shed any light on this? Is this a bug in PHP?

Kind regards,


Chris Neale
Contract Developer
Somerfield Stores Ltd.
 
If you are not the intended recipient of this e-mail, please preserve the
confidentiality of it and advise the sender immediately of any error in
transmission. Any disclosure, copying, distribution or action taken, or
omitted to be taken, by an unauthorised recipient in reliance upon the
contents of this e-mail is prohibited. Somerfield cannot accept liability
for any damage which you may sustain as a result of software viruses so
please carry out your own virus checks before opening an attachment. In
replying to this e-mail you are granting the right for that reply to be
forwarded to any other individual within the business and also to be read by
others. Any views expressed by an individual within this message do not
necessarily reflect the views of Somerfield.  Somerfield reserves the right
to intercept, monitor and record communications for lawful business
purposes.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory Leak... tracking?

[Robert Cummings]

I have written an extension that I compile into PHP in both CGI and
in module mode. The CGI version never crashes with a segfault in
my code; however, the module version almost always crashes on the
second page load. I've tracked it down to what appears to be one of
my pointers being overwritten, however, I have no way of finding out
where it is being overwritten, and Electric Fence bailed with a
free called on non malloc'd data error before the script even got
executed. Anyone know of another way I can track a buffer overflow
or underflow in PHP core code?

Cheers,
Rob.
-- 
.-.
| Robert Cummings |
:-`.
| Webdeployer - Chief PHP and Java Programmer  |
:--:
| Mail  : mailto:[EMAIL PROTECTED] |
| Phone : (613) 731-4046 x.109 |
:--:
| Website : http://www.webmotion.com   |
| Fax : (613) 260-9545 |
`--'

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] memory leak with mcrypt 2.4.5

[Remco Chang]

i found out that the mcrypt functions/library leak memory (libmcrypt 2.4.5)
with PHP 4.0.2 (on Solaris 2.6) and Apache 1.3.12 (actually, it's Stronghold
3.0).

the function that leaks is mcrypt_generic_init().

the PHP manual states that if you call mcrypt_generic_end (), it will clear
up the buffer and close the library.  however, i found out that the function
unfortunately does not dealloc the memory.

if someone could confirm this, i'll submit it into the bugs report.

remco chang
www.bountyquest.com


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]