Re: [Mojolicious] Reducing daemon memory footprint?
On Wed, 26 Sep 2018, sri wrote: > If that's the case, why not just fatpack your mojo app? Um, because I didn't know about fatpack? :-) Thanks, looks a good suggestion. The system will continue to be slow, but not as slow...
Re: [Mojolicious] Reducing daemon memory footprint?
Am Mittwoch, 26. September 2018 18:28:10 UTC+2 schrieb Charlie Brady: > > > This is why I need to run a persistent daemon rather than use CGI mode. > > root@jy-cx:~# time perl -e exit > > real0m0.035s > user0m0.000s > sys 0m0.030s > root@jy-cx:~# time perl -MMojolicious::Lite -e exit > > real0m11.011s > user0m8.690s > sys 0m0.400s > root@jy-cx:~# > > Looking at an strace there doesn't seem to be a single culprit for the > slow startup - there's just a lot of files to find, open, read, parse. > If that's the case, why not just fatpack your mojo app? -- sebastian -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Reducing daemon memory footprint?
This is why I need to run a persistent daemon rather than use CGI mode.
root@jy-cx:~# time perl -e exit
real0m0.035s
user0m0.000s
sys 0m0.030s
root@jy-cx:~# time perl -MMojolicious::Lite -e exit
real0m11.011s
user0m8.690s
sys 0m0.400s
root@jy-cx:~#
Looking at an strace there doesn't seem to be a single culprit for the
slow startup - there's just a lot of files to find, open, read, parse.
root@jy-cx:~# grep open ml.trace | wc -l
211
root@jy-cx:~#
...
root@jy-cx:~# grep open ml.trace | grep 12:21:26 | wc -l
16
root@jy-cx:~# grep open ml.trace | grep 12:21:27 | wc -l
28
root@jy-cx:~# grep open ml.trace | grep 12:21:28 | wc -l
16
root@jy-cx:~# grep open ml.trace | grep 12:21:29 | wc -l
19
root@jy-cx:~# grep open ml.trace | grep 12:21:30 | wc -l
13
root@jy-cx:~# grep open ml.trace | grep 12:21:31 | wc -l
2
root@jy-cx:~# grep open ml.trace | grep 12:21:32 | wc -l
15
root@jy-cx:~# grep open ml.trace | grep 12:21:33 | wc -l
8
root@jy-cx:~# grep open ml.trace | grep 12:21:34 | wc -l
18
root@jy-cx:~# grep open ml.trace | grep 12:21:35 | wc -l
24
root@jy-cx:~# grep open ml.trace | grep 12:21:36 | wc -l
8
root@jy-cx:~# grep open ml.trace | grep 12:21:37 | wc -l
19
root@jy-cx:~# grep open ml.trace | grep 12:21:38 | wc -l
11
root@jy-cx:~# grep open ml.trace | grep 12:21:39 | wc -l
14
root@jy-cx:~# grep open ml.trace | grep 12:21:40 | wc -l
0
root@jy-cx:~#
...
Initialising Time::HiRes::HiRes seems to be the single individual
parse/init step:
12:21:30.734184 open("/usr/lib/perl/5.22.0/Pod/Simple/BlackBox.pm",
O_RDONLY|O_L
ARGEFILE) = 16
12:21:31.607709 open("/usr/lib/perl/5.22.0/Time/HiRes.pm",
O_RDONLY|O_LARGEFILE) = 12
12:21:31.643254 open("/usr/lib/perl/5.22.0//auto/Time/HiRes/HiRes.so",
O_RDONLY|O_CLOEXEC) = 12
12:21:32.356811 open("/usr/lib/perl/vendor_perl/5.22.0/Mojo/Log.pm",
O_RDONLY|O_LARGEFILE) = 7
On Thu, 20 Sep 2018, Charlie Brady wrote:
>
> On Thu, 20 Sep 2018, Stefan Adams wrote:
>
> > Wow! That's fascinating! Thanks for sharing! I'm surprised by how
> > much the memory footprint grows for a minimal perl program to one
> > running M::L!
>
> Bear in mind this is one M::L program which has loaded (via the Mount
> plugin) a few dozen other M::L programs and then run for a few months
> (mostly idle).
>
> FWIW, this is on a PPC processor (perl 5.22.0, Mojolicious 7.33).
>
> > On Thu, Sep 20, 2018 at 2:57 PM Charlie Brady <
> > charlieb-m...@budge.apana.org.au> wrote:
> >
> > >
> > > Hi Stefan, thanks for trying to help.
> > >
> > > I know the perl executable will make some contribution to the memory
> > > footprint, but not much.
> > >
> > > I can measure that separately by running a minimum perl program and
> > > looking at its resource usage. If I run 'perl -e "<>"', I see this in
> > > /proc/$pid/status:
> > >
> > > ...
> > > VmPeak: 5476 kB
> > > VmSize: 5476 kB
> > > VmLck: 0 kB
> > > VmPin: 0 kB
> > > VmHWM: 3184 kB
> > > VmRSS: 3184 kB
> > > VmData: 468 kB
> > > VmStk: 132 kB
> > > VmExe: 4 kB
> > > VmLib: 4232 kB
> > > VmPTE:24 kB
> > > VmPMD: 0 kB
> > > VmSwap:0 kB
> > > Threads:1
> > > ...
> > >
> > > The same data from my M::L daemon is:
> > >
> > > ...
> > > VmPeak: 135548 kB
> > > VmSize: 133540 kB
> > > VmLck: 0 kB
> > > VmPin: 0 kB
> > > VmHWM: 67016 kB
> > > VmRSS: 65988 kB
> > > VmData:86940 kB
> > > VmStk: 176 kB
> > > VmExe: 4 kB
> > > VmLib: 31664 kB
> > > VmPTE: 132 kB
> > > VmPMD: 0 kB
> > > VmSwap:0 kB
> > > Threads:6
> > > ...
> > >
> > > CPAN has a few modules which might help gather some stats
> > > (Memory::Process, Memory::Stats, Memory::Usage). '-Dm' flag to perl might
> > > help too, if perl is compiled with DEBUG enabled.
> > >
> > > I notice that I can disable Mojolicious response cacheing; it'll be
> > > interesting to see what difference that makes.
> > >
> > > On Wed, 19 Sep 2018, Stefan Adams wrote:
> > >
> > > > I shouldn't involve myself in this topic because I haven't the
> > > > knowledge,
> > > > but it sounds fascinating. Wouldn't the majority of the 50MB memory
> > > > footprint be from the Perl interpreter itself? Wouldn't the question be,
> > > > how to reduce the size of the Perl interpreter? Different compile
> > > options?
> > > > e.g. I remember once compiling a "minimal" perl that was 2MB. I imagine
> > > > that would probably be insufficient to run Mojolicious, though I really
> > > > haven't a clue, but it suggests to me that it's possible to reduce the
> > > size
> > > > of the perl interpreter and, therefore it seems, the size of the
> > > > Mojolicious daemon?
> > > >
> > > > On Wed, Sep 19, 2018, 8:19 PM Charlie Brady
Re: [Mojolicious] Reducing daemon memory footprint?
On Thu, 20 Sep 2018, Stefan Adams wrote: > Wow! That's fascinating! Thanks for sharing! I'm surprised by how > much the memory footprint grows for a minimal perl program to one > running M::L! Bear in mind this is one M::L program which has loaded (via the Mount plugin) a few dozen other M::L programs and then run for a few months (mostly idle). FWIW, this is on a PPC processor (perl 5.22.0, Mojolicious 7.33). > On Thu, Sep 20, 2018 at 2:57 PM Charlie Brady < > charlieb-m...@budge.apana.org.au> wrote: > > > > > Hi Stefan, thanks for trying to help. > > > > I know the perl executable will make some contribution to the memory > > footprint, but not much. > > > > I can measure that separately by running a minimum perl program and > > looking at its resource usage. If I run 'perl -e "<>"', I see this in > > /proc/$pid/status: > > > > ... > > VmPeak: 5476 kB > > VmSize: 5476 kB > > VmLck: 0 kB > > VmPin: 0 kB > > VmHWM: 3184 kB > > VmRSS: 3184 kB > > VmData: 468 kB > > VmStk: 132 kB > > VmExe: 4 kB > > VmLib: 4232 kB > > VmPTE:24 kB > > VmPMD: 0 kB > > VmSwap:0 kB > > Threads:1 > > ... > > > > The same data from my M::L daemon is: > > > > ... > > VmPeak: 135548 kB > > VmSize: 133540 kB > > VmLck: 0 kB > > VmPin: 0 kB > > VmHWM: 67016 kB > > VmRSS: 65988 kB > > VmData:86940 kB > > VmStk: 176 kB > > VmExe: 4 kB > > VmLib: 31664 kB > > VmPTE: 132 kB > > VmPMD: 0 kB > > VmSwap:0 kB > > Threads:6 > > ... > > > > CPAN has a few modules which might help gather some stats > > (Memory::Process, Memory::Stats, Memory::Usage). '-Dm' flag to perl might > > help too, if perl is compiled with DEBUG enabled. > > > > I notice that I can disable Mojolicious response cacheing; it'll be > > interesting to see what difference that makes. > > > > On Wed, 19 Sep 2018, Stefan Adams wrote: > > > > > I shouldn't involve myself in this topic because I haven't the knowledge, > > > but it sounds fascinating. Wouldn't the majority of the 50MB memory > > > footprint be from the Perl interpreter itself? Wouldn't the question be, > > > how to reduce the size of the Perl interpreter? Different compile > > options? > > > e.g. I remember once compiling a "minimal" perl that was 2MB. I imagine > > > that would probably be insufficient to run Mojolicious, though I really > > > haven't a clue, but it suggests to me that it's possible to reduce the > > size > > > of the perl interpreter and, therefore it seems, the size of the > > > Mojolicious daemon? > > > > > > On Wed, Sep 19, 2018, 8:19 PM Charlie Brady < > > > charlieb-m...@budge.apana.org.au> wrote: > > > > > > > > > > > Some Mojolicious::Lite code I've written has been adapted by another > > group > > > > in my company to run on their embedded controller (limited CPU and > > limited > > > > memory). They are now asking for my advice on how to reduce the memory > > > > footprint of their daemon. It's consuming about 50MB at the moment. > > > > > > > > Does anyone have some advice on where to start looking at where memory > > is > > > > being used and how to minimise it? > > > > > > > > Thanks > > > > > > > > > > > > > >
Re: [Mojolicious] Reducing daemon memory footprint?
Wow! That's fascinating! Thanks for sharing! I'm surprised by how much the memory footprint grows for a minimal perl program to one running M::L! On Thu, Sep 20, 2018 at 2:57 PM Charlie Brady < charlieb-m...@budge.apana.org.au> wrote: > > Hi Stefan, thanks for trying to help. > > I know the perl executable will make some contribution to the memory > footprint, but not much. > > I can measure that separately by running a minimum perl program and > looking at its resource usage. If I run 'perl -e "<>"', I see this in > /proc/$pid/status: > > ... > VmPeak: 5476 kB > VmSize: 5476 kB > VmLck: 0 kB > VmPin: 0 kB > VmHWM: 3184 kB > VmRSS: 3184 kB > VmData: 468 kB > VmStk: 132 kB > VmExe: 4 kB > VmLib: 4232 kB > VmPTE:24 kB > VmPMD: 0 kB > VmSwap:0 kB > Threads:1 > ... > > The same data from my M::L daemon is: > > ... > VmPeak: 135548 kB > VmSize: 133540 kB > VmLck: 0 kB > VmPin: 0 kB > VmHWM: 67016 kB > VmRSS: 65988 kB > VmData:86940 kB > VmStk: 176 kB > VmExe: 4 kB > VmLib: 31664 kB > VmPTE: 132 kB > VmPMD: 0 kB > VmSwap:0 kB > Threads:6 > ... > > CPAN has a few modules which might help gather some stats > (Memory::Process, Memory::Stats, Memory::Usage). '-Dm' flag to perl might > help too, if perl is compiled with DEBUG enabled. > > I notice that I can disable Mojolicious response cacheing; it'll be > interesting to see what difference that makes. > > On Wed, 19 Sep 2018, Stefan Adams wrote: > > > I shouldn't involve myself in this topic because I haven't the knowledge, > > but it sounds fascinating. Wouldn't the majority of the 50MB memory > > footprint be from the Perl interpreter itself? Wouldn't the question be, > > how to reduce the size of the Perl interpreter? Different compile > options? > > e.g. I remember once compiling a "minimal" perl that was 2MB. I imagine > > that would probably be insufficient to run Mojolicious, though I really > > haven't a clue, but it suggests to me that it's possible to reduce the > size > > of the perl interpreter and, therefore it seems, the size of the > > Mojolicious daemon? > > > > On Wed, Sep 19, 2018, 8:19 PM Charlie Brady < > > charlieb-m...@budge.apana.org.au> wrote: > > > > > > > > Some Mojolicious::Lite code I've written has been adapted by another > group > > > in my company to run on their embedded controller (limited CPU and > limited > > > memory). They are now asking for my advice on how to reduce the memory > > > footprint of their daemon. It's consuming about 50MB at the moment. > > > > > > Does anyone have some advice on where to start looking at where memory > is > > > being used and how to minimise it? > > > > > > Thanks > > > > > > > > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Reducing daemon memory footprint?
There's some good general advice from our friend Brian D Foy here: https://stackoverflow.com/questions/9733146/tips-for-keeping-perl-memory-usage-low To bring this more closely focussed on Mojolicious, my daemon is loading a ew dozen other Mojolicious::Lite applications using the Mount plugin. And all of those are using a common plugin. I'm wondering how much duplication there is within the sandboxing of each mounted application, and what might be possible to optimize the sharing of data/compiled code between the instances. On Thu, 20 Sep 2018, Charlie Brady wrote: > > Hi Stefan, thanks for trying to help. > > I know the perl executable will make some contribution to the memory > footprint, but not much. > > I can measure that separately by running a minimum perl program and > looking at its resource usage. If I run 'perl -e "<>"', I see this in > /proc/$pid/status: > > ... > VmPeak: 5476 kB > VmSize: 5476 kB > VmLck: 0 kB > VmPin: 0 kB > VmHWM: 3184 kB > VmRSS: 3184 kB > VmData: 468 kB > VmStk: 132 kB > VmExe: 4 kB > VmLib: 4232 kB > VmPTE:24 kB > VmPMD: 0 kB > VmSwap:0 kB > Threads:1 > ... > > The same data from my M::L daemon is: > > ... > VmPeak: 135548 kB > VmSize: 133540 kB > VmLck: 0 kB > VmPin: 0 kB > VmHWM: 67016 kB > VmRSS: 65988 kB > VmData:86940 kB > VmStk: 176 kB > VmExe: 4 kB > VmLib: 31664 kB > VmPTE: 132 kB > VmPMD: 0 kB > VmSwap:0 kB > Threads:6 > ... > > CPAN has a few modules which might help gather some stats > (Memory::Process, Memory::Stats, Memory::Usage). '-Dm' flag to perl might > help too, if perl is compiled with DEBUG enabled. > > I notice that I can disable Mojolicious response cacheing; it'll be > interesting to see what difference that makes. > > On Wed, 19 Sep 2018, Stefan Adams wrote: > > > I shouldn't involve myself in this topic because I haven't the knowledge, > > but it sounds fascinating. Wouldn't the majority of the 50MB memory > > footprint be from the Perl interpreter itself? Wouldn't the question be, > > how to reduce the size of the Perl interpreter? Different compile options? > > e.g. I remember once compiling a "minimal" perl that was 2MB. I imagine > > that would probably be insufficient to run Mojolicious, though I really > > haven't a clue, but it suggests to me that it's possible to reduce the size > > of the perl interpreter and, therefore it seems, the size of the > > Mojolicious daemon? > > > > On Wed, Sep 19, 2018, 8:19 PM Charlie Brady < > > charlieb-m...@budge.apana.org.au> wrote: > > > > > > > > Some Mojolicious::Lite code I've written has been adapted by another group > > > in my company to run on their embedded controller (limited CPU and limited > > > memory). They are now asking for my advice on how to reduce the memory > > > footprint of their daemon. It's consuming about 50MB at the moment. > > > > > > Does anyone have some advice on where to start looking at where memory is > > > being used and how to minimise it? > > > > > > Thanks > > > > > > > >
Re: [Mojolicious] Reducing daemon memory footprint?
Hi Stefan, thanks for trying to help. I know the perl executable will make some contribution to the memory footprint, but not much. I can measure that separately by running a minimum perl program and looking at its resource usage. If I run 'perl -e "<>"', I see this in /proc/$pid/status: ... VmPeak: 5476 kB VmSize: 5476 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 3184 kB VmRSS: 3184 kB VmData: 468 kB VmStk: 132 kB VmExe: 4 kB VmLib: 4232 kB VmPTE:24 kB VmPMD: 0 kB VmSwap:0 kB Threads:1 ... The same data from my M::L daemon is: ... VmPeak: 135548 kB VmSize: 133540 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 67016 kB VmRSS: 65988 kB VmData:86940 kB VmStk: 176 kB VmExe: 4 kB VmLib: 31664 kB VmPTE: 132 kB VmPMD: 0 kB VmSwap:0 kB Threads:6 ... CPAN has a few modules which might help gather some stats (Memory::Process, Memory::Stats, Memory::Usage). '-Dm' flag to perl might help too, if perl is compiled with DEBUG enabled. I notice that I can disable Mojolicious response cacheing; it'll be interesting to see what difference that makes. On Wed, 19 Sep 2018, Stefan Adams wrote: > I shouldn't involve myself in this topic because I haven't the knowledge, > but it sounds fascinating. Wouldn't the majority of the 50MB memory > footprint be from the Perl interpreter itself? Wouldn't the question be, > how to reduce the size of the Perl interpreter? Different compile options? > e.g. I remember once compiling a "minimal" perl that was 2MB. I imagine > that would probably be insufficient to run Mojolicious, though I really > haven't a clue, but it suggests to me that it's possible to reduce the size > of the perl interpreter and, therefore it seems, the size of the > Mojolicious daemon? > > On Wed, Sep 19, 2018, 8:19 PM Charlie Brady < > charlieb-m...@budge.apana.org.au> wrote: > > > > > Some Mojolicious::Lite code I've written has been adapted by another group > > in my company to run on their embedded controller (limited CPU and limited > > memory). They are now asking for my advice on how to reduce the memory > > footprint of their daemon. It's consuming about 50MB at the moment. > > > > Does anyone have some advice on where to start looking at where memory is > > being used and how to minimise it? > > > > Thanks > > > >
Re: [Mojolicious] Reducing daemon memory footprint?
I shouldn't involve myself in this topic because I haven't the knowledge, but it sounds fascinating. Wouldn't the majority of the 50MB memory footprint be from the Perl interpreter itself? Wouldn't the question be, how to reduce the size of the Perl interpreter? Different compile options? e.g. I remember once compiling a "minimal" perl that was 2MB. I imagine that would probably be insufficient to run Mojolicious, though I really haven't a clue, but it suggests to me that it's possible to reduce the size of the perl interpreter and, therefore it seems, the size of the Mojolicious daemon? On Wed, Sep 19, 2018, 8:19 PM Charlie Brady < charlieb-m...@budge.apana.org.au> wrote: > > Some Mojolicious::Lite code I've written has been adapted by another group > in my company to run on their embedded controller (limited CPU and limited > memory). They are now asking for my advice on how to reduce the memory > footprint of their daemon. It's consuming about 50MB at the moment. > > Does anyone have some advice on where to start looking at where memory is > being used and how to minimise it? > > Thanks > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
[Mojolicious] Reducing daemon memory footprint?
Some Mojolicious::Lite code I've written has been adapted by another group in my company to run on their embedded controller (limited CPU and limited memory). They are now asking for my advice on how to reduce the memory footprint of their daemon. It's consuming about 50MB at the moment. Does anyone have some advice on where to start looking at where memory is being used and how to minimise it? Thanks
