Re: perl's memory leak
On Fri, 8 Dec 2000, Perrin Harkins wrote: Unfortunately, GTop is kind of a pain to compile. It seems to depend on some Gnome stuff. We use Apache::SizeLimit for this reason, and it works well. there's a configure --disable-gnome switch for libgtop. can still be a pain though.
Re: perl's memory leak
The long time mythical use less qw(memory); pragma was always intended to address issues like that. It would be fairly straightforward to implement (set a flag like use strict and check it wherever memory could usefully be freed). Tim. On Sat, Dec 09, 2000 at 07:53:38PM +0300, Ivan E. Panchenko wrote: No, i did not mean freeing memory from lexicals. I meant the memory allocated for the temporary results, such as my $a = 'x' x 100 Here perl allocates 1M for $a and 1M for evaluating the right part, after that it is possible to undef $a and reuse its memory (1M), but the right part memory (one more 1M) can be used nowhere except the same line of code. This is strange. I understand that it can be an optimization trick, but when you frequently operate with megabytes, this becomes a pleasant feature which looks and behaves like a memory leak :) Ivan The matter is that perl DOES NOT REUSE MEMORY allocated for intermediate calculation results. This is specially harmful to data-intensive modperl applications where one perl process processes many queries and can leak great amount of memory. This is known and it's not really a leak. Perl knows about that memory and does re-use it, the next time it needs that lexical variable. It's a performance optimization. Try running your code multiple times and you should see memory stay at the same level after the first run. There has been discussion about this on the mailing list which you can find in the archives. There has also been talk about changing this behavior for mod_perl 2, which Doug is working on. Anyway, if you just want the memory back, undef your lexicals after you finish with them. - Perrin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
No, i did not mean freeing memory from lexicals. I meant the memory allocated for the temporary results, such as my $a = 'x' x 100 Here perl allocates 1M for $a and 1M for evaluating the right part, after that it is possible to undef $a and reuse its memory (1M), but the right part memory (one more 1M) can be used nowhere except the same line of code. This is strange. I understand that it can be an optimization trick, but when you frequently operate with megabytes, this becomes a pleasant feature which looks and behaves like a memory leak :) Ivan The matter is that perl DOES NOT REUSE MEMORY allocated for intermediate calculation results. This is specially harmful to data-intensive modperl applications where one perl process processes many queries and can leak great amount of memory. This is known and it's not really a leak. Perl knows about that memory and does re-use it, the next time it needs that lexical variable. It's a performance optimization. Try running your code multiple times and you should see memory stay at the same level after the first run. There has been discussion about this on the mailing list which you can find in the archives. There has also been talk about changing this behavior for mod_perl 2, which Doug is working on. Anyway, if you just want the memory back, undef your lexicals after you finish with them. - Perrin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
it seems as if most (if not all) the techniques for checking the size of the current process are _very_ platform specific. on linux you can use Apache::SizeLimit::linux_size_check which is just parsing /proc/self/status. - mark Stas Bekman wrote: On Thu, 7 Dec 2000 [EMAIL PROTECTED] wrote: The output I get is used memory = 0 used memory = 0 used memory = 0 used memory = 0 used memory = 0 I get the same under perl 5.6.0 on linux, looks like BSD::Resource doesn't work there :( Anyone? Use gtop instead (if you have it): use GTop (); print GTop-new-proc_mem($$)-size,"\n"; [snip] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
On Fri, 8 Dec 2000, mark warren bracher wrote: it seems as if most (if not all) the techniques for checking the size of the current process are _very_ platform specific. on linux you can use Apache::SizeLimit::linux_size_check If you have linux you have (or can have GTop), which gives you an API to do this and many other things. Apache::SizeLimit::linux_size_check is just a custom function that you cannot really re-use (unless you put it into some other module... which is just parsing /proc/self/status. - mark Stas Bekman wrote: On Thu, 7 Dec 2000 [EMAIL PROTECTED] wrote: The output I get is used memory = 0 used memory = 0 used memory = 0 used memory = 0 used memory = 0 I get the same under perl 5.6.0 on linux, looks like BSD::Resource doesn't work there :( Anyone? Use gtop instead (if you have it): use GTop (); print GTop-new-proc_mem($$)-size,"\n"; [snip] _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://logilune.com/ http://singlesheaven.com http://perl.apache.org http://perlmonth.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
On Fri, 8 Dec 2000, Stas Bekman wrote: If you have linux you have (or can have GTop), which gives you an API to do this and many other things. Apache::SizeLimit::linux_size_check is just a custom function that you cannot really re-use (unless you put it into some other module... Unfortunately, GTop is kind of a pain to compile. It seems to depend on some Gnome stuff. We use Apache::SizeLimit for this reason, and it works well. - Perrin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
perl's memory leak
Today I discovered a strange behaiviour of perl,
and I wonder if anybody can tell me what to do with it.
The matter is that perl DOES NOT REUSE MEMORY allocated for
intermediate calculation results. This is specially harmful to
data-intensive modperl applications where one perl process processes
many queries and can leak great amount of memory.
The example is below:
use BSD::Resource;
my $cc = 'a' x 2000 ;# alocates 20Mb for the right part and
# 20Mb for $a
p;
{ my $a = $cc.'x'; # allocates 20 more Mb for right part
# and 20 for a
p;
undef $a; # deallocates $a
}
p;
{ my $b = $cc.'y'; # allocates 20 more Mb for right part
# and reuses deallocated 20Mb for b
p;
undef $b;
}
p;
sub p {
print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
# end of example.
Output:
used memory = 40772
used memory = 79804
used memory = 80068
used memory = 99676
used memory = 99700
##
Here I used BSD:Resource to measure consumed memory. Its result seems to
be correlated with the amount of memory taken by the process from the OS.
#
This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
Same things where noticed on Linux and probably on Solaris too.
Ivan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
The output I get is
used memory = 0
used memory = 0
used memory = 0
used memory = 0
used memory = 0
I'm interested in how many leaks are possible in mod_perl
though because my mod_perl processes are getting bigger
with time -- about 200 requests is making the process
fatter by 1mb on the average. I'm watching to see if
they will max out around the current level of 10 mb per child.
On Thu, Dec 07, 2000 at 07:53:16PM +0300, Ivan E. Panchenko wrote:
Today I discovered a strange behaiviour of perl,
and I wonder if anybody can tell me what to do with it.
The matter is that perl DOES NOT REUSE MEMORY allocated for
intermediate calculation results. This is specially harmful to
data-intensive modperl applications where one perl process processes
many queries and can leak great amount of memory.
The example is below:
use BSD::Resource;
my $cc = 'a' x 2000 ;# alocates 20Mb for the right part and
# 20Mb for $a
p;
{ my $a = $cc.'x'; # allocates 20 more Mb for right part
# and 20 for a
p;
undef $a;# deallocates $a
}
p;
{ my $b = $cc.'y'; # allocates 20 more Mb for right part
# and reuses deallocated 20Mb for b
p;
undef $b;
}
p;
sub p {
print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
# end of example.
Output:
used memory = 40772
used memory = 79804
used memory = 80068
used memory = 99676
used memory = 99700
##
Here I used BSD:Resource to measure consumed memory. Its result seems to
be correlated with the amount of memory taken by the process from the OS.
#
This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
Same things where noticed on Linux and probably on Solaris too.
Ivan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
On Thu, 7 Dec 2000, Ivan E. Panchenko wrote:
Today I discovered a strange behaiviour of perl,
and I wonder if anybody can tell me what to do with it.
The matter is that perl DOES NOT REUSE MEMORY allocated for
intermediate calculation results. This is specially harmful to
data-intensive modperl applications where one perl process processes
many queries and can leak great amount of memory.
When you will rerun the same code again (the same request) in the same
httpd process, the memory won't grow. There is no leak, that's the way
Perl optimizes memory allocation.
On the contrary, when you will rerun the same code in the same process,
it'll be executed even faster since the memory allocation for lexically
scoped variables $a and $b was done in the first execution. So even if you
undef them the memory is still allocated.
You can easily test it with httpd -X. See more examples of this kind of
testing with gtop and Devel::Peek in the performance chapter in the guide.
Of course if you don't like it, you can complain to p5p list :)
The example is below:
use BSD::Resource;
my $cc = 'a' x 2000 ;# alocates 20Mb for the right part and
# 20Mb for $a
p;
{ my $a = $cc.'x'; # allocates 20 more Mb for right part
# and 20 for a
p;
undef $a;# deallocates $a
}
p;
{ my $b = $cc.'y'; # allocates 20 more Mb for right part
# and reuses deallocated 20Mb for b
p;
undef $b;
}
p;
sub p {
print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
# end of example.
Output:
used memory = 40772
used memory = 79804
used memory = 80068
used memory = 99676
used memory = 99700
##
Here I used BSD:Resource to measure consumed memory. Its result seems to
be correlated with the amount of memory taken by the process from the OS.
#
This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
Same things where noticed on Linux and probably on Solaris too.
Ivan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://logilune.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
On Thu, 7 Dec 2000, Ivan E. Panchenko wrote: Today I discovered a strange behaiviour of perl, and I wonder if anybody can tell me what to do with it. The matter is that perl DOES NOT REUSE MEMORY allocated for intermediate calculation results. This is specially harmful to data-intensive modperl applications where one perl process processes many queries and can leak great amount of memory. This is known and it's not really a leak. Perl knows about that memory and does re-use it, the next time it needs that lexical variable. It's a performance optimization. Try running your code multiple times and you should see memory stay at the same level after the first run. There has been discussion about this on the mailing list which you can find in the archives. There has also been talk about changing this behavior for mod_perl 2, which Doug is working on. Anyway, if you just want the memory back, undef your lexicals after you finish with them. - Perrin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[OT] Re: perl's memory leak
Hi Ivan,
It's not really mod_perl, but is relevant to people on the list I guess...
If you really play aorund with this, you'll find some interesting variations.
If I assign $cc using a for loop
my $c;
for ( 1..2000) { $cc .= 'a'; }
it's a lot slower, but only uses half as much ram
also if I do the assignment first and then add a char:
my $a = $cc; # allocates 20 more Mb for right part
p( "assigned a" );
$a .= 'x';
p( "changed a" );
undef $a; # deallocates $a
p( "undefed a" );
my $b = $cc; # allocates 20 more Mb for right part
$b .= 'a';
# and reuses deallocated 20Mb for b
p( "defined b" );
undef $b;
p( "undefed b");
sub p {
my $mesg = shift;
print "$mesg used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
I get:
assigned a used memory = 40212
changed a used memory = 40212
undefed a used memory = 40212
defined b used memory = 40712
undefed b used memory = 40712
Which is more what you would hope to see right?
andrew
On Thu, 07 Dec 2000, Ivan E. Panchenko wrote:
Today I discovered a strange behaiviour of perl,
and I wonder if anybody can tell me what to do with it.
The matter is that perl DOES NOT REUSE MEMORY allocated for
intermediate calculation results. This is specially harmful to
data-intensive modperl applications where one perl process processes
many queries and can leak great amount of memory.
The example is below:
use BSD::Resource;
my $cc = 'a' x 2000 ;# alocates 20Mb for the right part and
# 20Mb for $a
p;
{ my $a = $cc.'x'; # allocates 20 more Mb for right part
# and 20 for a
p;
undef $a;# deallocates $a
}
p;
{ my $b = $cc.'y'; # allocates 20 more Mb for right part
# and reuses deallocated 20Mb for b
p;
undef $b;
}
p;
sub p {
print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
# end of example.
Output:
used memory = 40772
used memory = 79804
used memory = 80068
used memory = 99676
used memory = 99700
##
Here I used BSD:Resource to measure consumed memory. Its result seems to
be correlated with the amount of memory taken by the process from the OS.
#
This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
Same things where noticed on Linux and probably on Solaris too.
Ivan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
...
Andrew Wyllie [EMAIL PROTECTED]Open Source Integrator
v.206.729.7439 __We can catify or stringify,
c.206.851.9876separately or together!__ perl-5.005_03
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
On Thu, 7 Dec 2000 [EMAIL PROTECTED] wrote:
The output I get is
used memory = 0
used memory = 0
used memory = 0
used memory = 0
used memory = 0
I get the same under perl 5.6.0 on linux, looks like BSD::Resource doesn't
work there :( Anyone?
Use gtop instead (if you have it):
use GTop ();
print GTop-new-proc_mem($$)-size,"\n";
I'm interested in how many leaks are possible in mod_perl
though because my mod_perl processes are getting bigger
with time -- about 200 requests is making the process
fatter by 1mb on the average. I'm watching to see if
they will max out around the current level of 10 mb per child.
On Thu, Dec 07, 2000 at 07:53:16PM +0300, Ivan E. Panchenko wrote:
Today I discovered a strange behaiviour of perl,
and I wonder if anybody can tell me what to do with it.
The matter is that perl DOES NOT REUSE MEMORY allocated for
intermediate calculation results. This is specially harmful to
data-intensive modperl applications where one perl process processes
many queries and can leak great amount of memory.
The example is below:
use BSD::Resource;
my $cc = 'a' x 2000 ;# alocates 20Mb for the right part and
# 20Mb for $a
p;
{ my $a = $cc.'x'; # allocates 20 more Mb for right part
# and 20 for a
p;
undef $a; # deallocates $a
}
p;
{ my $b = $cc.'y'; # allocates 20 more Mb for right part
# and reuses deallocated 20Mb for b
p;
undef $b;
}
p;
sub p {
print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
# end of example.
Output:
used memory = 40772
used memory = 79804
used memory = 80068
used memory = 99676
used memory = 99700
##
Here I used BSD:Resource to measure consumed memory. Its result seems to
be correlated with the amount of memory taken by the process from the OS.
#
This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
Same things where noticed on Linux and probably on Solaris too.
Ivan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://logilune.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: perl's memory leak
You probably tried this script on linux or some other not fully BSD
compartible system. We obtained same zeros on linux, where getrusage()
means something else than on FreeBSD,
but if you try measuring memory sizes with ps or top, you should
observe the mentioned leak. Please insert sleep(10) after print STDERR
and watch "top".
Ivan
On Thu, 7 Dec 2000 [EMAIL PROTECTED] wrote:
The output I get is
used memory = 0
used memory = 0
used memory = 0
used memory = 0
used memory = 0
I'm interested in how many leaks are possible in mod_perl
though because my mod_perl processes are getting bigger
with time -- about 200 requests is making the process
fatter by 1mb on the average. I'm watching to see if
they will max out around the current level of 10 mb per child.
On Thu, Dec 07, 2000 at 07:53:16PM +0300, Ivan E. Panchenko wrote:
Today I discovered a strange behaiviour of perl,
and I wonder if anybody can tell me what to do with it.
The matter is that perl DOES NOT REUSE MEMORY allocated for
intermediate calculation results. This is specially harmful to
data-intensive modperl applications where one perl process processes
many queries and can leak great amount of memory.
The example is below:
use BSD::Resource;
my $cc = 'a' x 2000 ;# alocates 20Mb for the right part and
# 20Mb for $a
p;
{ my $a = $cc.'x'; # allocates 20 more Mb for right part
# and 20 for a
p;
undef $a; # deallocates $a
}
p;
{ my $b = $cc.'y'; # allocates 20 more Mb for right part
# and reuses deallocated 20Mb for b
p;
undef $b;
}
p;
sub p {
print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
# end of example.
Output:
used memory = 40772
used memory = 79804
used memory = 80068
used memory = 99676
used memory = 99700
##
Here I used BSD:Resource to measure consumed memory. Its result seems to
be correlated with the amount of memory taken by the process from the OS.
#
This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
Same things where noticed on Linux and probably on Solaris too.
Ivan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
