On Sat, 2008-10-25 at 05:59 +0200, Michael Lackhoff wrote:
> On 24.10.2008 15:03 Michael Peters wrote:
>
> > This is only true if those structures were created during run time and go
> > out of scope at run time.
> > If they are generated at compile time or attached to global variables or
> > package level variables,
> > they will not be re-used by Perl.
> Wait a minute, I would like to do exactly that: use a config module in
> startup.pl that loads some massive config hashes in the hope that the
> memory they use will be shared:
I think there was some confusion here about what was being said.
Michael Peters' comment about memory reuse was saying that:
- if at runtime, you load a large memory structure
- then let those variables go out of scope
- then that memory will be come available for reuse by Perl
but
- if you load that same large memory structure at server startup
- and the variables don't go out of scope,
- then that memory stays used
>
> package MyConfig;
> our $aHugeConfigHash = load_data_from_config_file();
>
> (well sort of, it is actually wrapped in an accessor but that gets its
> data from the package variable)
>
> Are you saying, I cannot share the memory this way? And if so, is there
> an alternative?
I do exactly this - a lot of my website is config driven. At startup, I
load a large data structure (7,000 lines of YAML ~~ 700kB in memory),
that I can access from any module.
This data structure is considered by my app to be read-only, but not
enforced.
Plug: You may want to look at Config::Merge (my module for
loading a tree of YAML / JSON / XML / Perl / INI /
Config::General files into a single config hash:
http://search.cpan.org/~drtech/Config-Merge-1.00/Merge.pm
Similarly, I preload almost all the modules that I will use at compile
time. I exclude certain seldom used large modules, such as PDF::API2,
which is require'd when needed.
This makes forking a new child very fast, thanks to copy-on-write in
linux.
Unfortunately, in Perl, there is no way to make sure that
data-that-will-never-change and compiled code is stored on a separate
page from data-that-will-change. So, it is likely that, while your
config data starts out completely shared, over time, it will become less
so.
As Stas Bekman said in his Improving mod_perl Sites' Performance
article:
http://www.perl.com/pub/a/2002/07/30/mod_perl.html
don't aim to make MaxRequestsPerChild = 10000, as the amount of memory
each child consumes will just increase over time, while creating a new
child is cheap, thanks to your preloading.
You may want to look at Linux::Smaps
http://search.cpan.org/author/OPI/Linux-Smaps-0.06/lib/Linux/Smaps.pm
and at smem.pl (a script which uses Linux::Smaps to print out memory
usage of a process)
http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html
to get some idea of just how shared the memory in your httpd processes
are.
Some examples of smem.pl output from my live site: (compare shared vs
private clean + private dirty)
PARENT PROCESS:
---------------
VMSIZE: 135036 kb
RSS: 45784 kb total
36616 kb shared
1012 kb private clean
8156 kb private dirty
OLDEST CHILD:
-------------
VMSIZE: 143780 kb
RSS: 53140 kb total
33912 kb shared
12 kb private clean
19216 kb private dirty
YOUNGEST CHILD:
---------------
VMSIZE: 138052 kb
RSS: 47172 kb total
36272 kb shared
0 kb private clean
10900 kb private dirty
hth
Clint
