Re: How do I minimize memory usage with WSGI and Apache?

[Graham Dumpleton]

On Aug 11, 9:08 am, Jumpfroggy  wrote:
> On Aug 10, 2:24 pm, dartdog  wrote:
>
> > Big help!! Had just started getting using too much memory notices from
> > Webfaction yesterday evening...
>
> Yeah, I didn't even pay attention till I hit about 120-140MB on my
> 80MB plan and got everything turned off.  Otherwise I would have just
> let it grow, not paying attention.
>
> > The other item is using the crontab -e command from ssh to comment out
> > apps you don't need to have running For some reason could not find
> > that when I was looking for it and one of my sites just kept
> > restarting after .stop, which I understand but did not know how to
> > stop the behavior!
>
> Good catch!  I noticed that they restart themselves, didn't bother
> finding out why.
>
> Here's my complete writeup and 
> howto:http://forum.webfaction.com/viewtopic.php?pid=11096
>
> Thanks to those in this thread for helping me make this work.

FWIW, WebFaction is providing mod_wsgi 2.5 now. Got mail from Remi
about it today.

A few other comments about your post on WebFaction forums.

Don't use 'processes=1' with WSGIDaemonProcess directive. The default
is one process anyway, and using 'processes' option has side effect of
setting wsgi.multiprocess to be true even if there is only one
process. Yes, it may seem odd, but it is done that way to allow
multiprocess to be flagged as true when using a cluster of machines
with only one process on each server.

When inactivity timeout expires, Django is technically restarted, it
is actually shutdown. Although a new daemon process is started, that
process isn't preloaded with anything and so is effectively an empty
process. This is why on next request it has to load Django again. That
is, lazily loads Django only when known it is required.

I now understand what you meant about supervisor process. Though you
were talking about separate supervisord application. :-(

Only other thing is something I think I said before. That is that
MinSpareThreads and MaxSpareThreads only make sense when multiples of
ThreadsPerChild. This is because the number of threads in a process is
fixed, it will not shutdown or start up threads within a process.
These values are there used to govern number of active processes
between limits as set down by StartServers and MaxClients/
ThreadsPerChild. In other words, granularity is actually at process
level and not thread level.

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Graham Dumpleton]

On Aug 11, 3:11 am, Jumpfroggy  wrote:
> Wow, lots of good feedback here.  Thanks!
>
> On Aug 9, 6:17 pm, Graham Dumpleton 
> wrote:
>
> > Replace use of mod_python with mod_wsgi.
>
> I'm using mod_wsgi for everything already, forgot to say.  When I
> researched, it seemed like the newer/better way to do django.
>
> > Ensure you use prefork MPM
> > for Apache and delegate Django instances to run in mod_wsgi daemon
> > mode process.
> > Sorry, meant to say 'worker' rather than 'prefork'. DO NOT USE
> > PREFORK. Hmmm, that was a bad stuff up. :-)
>
> I just started using the worker MPM for one of my apps.  It didn't
> seem to give any performance or memory savings, since my app is so
> small/simple.  I'll keep an eye on it for later.

In WebFaction model possibly wouldn't as you tend to be running as
much as possible in single process anyway to keep within your memory
limits.

> Using daemon mode gives me auto-reloading by touching the .wsgi file
> (yay), but it uses more memory.  Instead of supervisor (3MB) and
> apache worker/WSGI/Django (16MB) processes, now I have supervisor
> (3MB), apache worker (3MB), and wsgi/django (15MB).  Not a huge gain,
> but just a bit more.  I'm willing to live with that for the auto-
> reloading benefit, however.

Why do you still need supervisor. Certainly not required for mod_wsgi.
Am a bit confused.

> > Having done that, you can use inactivity-timeout option
> > for mod_wsgi daemon processes to have the process restarted and
> > returned to low memory footprint idle process after no use for a
> > while.
>
> This is exactly what I wanted!!!  It's funny how when I finally read
> the description for something, it matches exactly what I'm looking
> for.  I just didn't know where to look.  I just tried it, and yes my
> memory baloons up to 16MB, then comes back down later.  The only
> strange thing is that I set it to 10s (very short) to test out, but it
> takes a while for the process to actually be reclaimed.  If I put
> inactivity-timeout=10, then load the page, then close the browser,
> memory usage should be reclaimed in 10 seconds, right?  It seems more
> like 1 minute wait, which is fine but strange to me.  Maybe there's
> only polling at certain intervals (checks every 1 minute), so the
> resolution isn't that fine.  Maybe it's dependent on the deadlock
> timeout?  Don't know.

Nothing to do with deadlock timeout. You should be able to set values
less than 1 minute. I wouldn't recommend to short a timeout though as
requests by same user within a browsing session are often reasonable
time apart. If too short a timeout, then Django gets kicked out
between requests and has to be reloaded every time, which would give
CGI like performance.

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Graham Dumpleton]

On Aug 11, 12:25 am, Dj Gilcrease  wrote:
> On Mon, Aug 10, 2009 at 6:43 AM, Graham
>
> Dumpleton wrote:
> > These values for min and max threads are a bit dubious because you
> > have a single process and that will have fixed 25 threads. Usually
> > these are defined in multiples of ThreadsPerChild and not less like
> > this. Possibly just  means the values are ignored.
>
> The MaxClients (according to the Apache docs is #MaxServers *
> #ThreadsPerChild) I just set it explicitly so I remember what it is,
> though it is actually ignored

It is not ignored. What you are describing is the fallback default
behaviour if the directives are not defined. If the directives are
defined they will take precedence. I would always add them all
explicitly.

> >> ThreadsPerChild 25
> >> MaxRequestsPerChild 5000
>
> > Don't see much point for MaxRequestsPerChild.
>
> This just restarts the child after 5k requests, which helps clear out
> some memory on occasion (I am still trying to get rrd or something run
> on on Webfaction so I can actually map memory usage over time, and
> start tuning Apache to my actual usage needs better)

But if using daemon mode, MaxRequestsPerChild only applies to the main
Apache server child processes. These are only serving up static files,
if you aren't getting WebFaction nginx front end to handle them, and
the task of proxying to mod_wsgi daemon processes. The processes
therefore shouldn't grow memory.

FWIW, if you are only running 5 threads in mod_wsgi daemon process and
static files handled by nginx front end anyway, you could drop
ThreadsPerChild to be a lower value and save more memory in main
Apache server child processes as stack memory for extra threads you
eliminate would no longer be allocated. Just remember to also drop
MaxClients to match if you want to keep it done to a single process
maximum, although if you don't I think Apache will complain that
calculated maximum servers is greater than ServerLimit.

> > If you application does leak Python objects due to inability to break
> > reference count cycles, there shouldn't really be a need for maximum
> > requests.
>
> Though my app does no leak, that I have noticed anyways, I figure a
> maximum-requests would be good just in case I suddenly get a flood of
> traffic, it will restart the daemon even if the hour of idle time has
> not passed.

The other one I forgot to mention is where specific requests have a
large transient memory usage. This will bump out memory usage and once
pushed out, doesn't come back.

The big danger with using multithreading is where a URL with large
transient memory requirement gets hit by multiple requests at the same
time.

I have been thinking for a while with a feature for mod_wsgi whereby
you could using a directive specify a limit as to how many concurrent
requests you want to allow to enter a specific URL or some subset of
URLs. This would help in limiting such memory usage explosions. The
only reason haven't added feature is that it could just as easily be
done in WSGI middleware, however, in that case people have to
implement it themselves and so more than likely they will not bother.

Other than such an ability, only workaround at moment is to use
maximum requests to recycle process often. In worst case may want to
split application across multiple daemon process groups and delegate
memory hungry URLs to daemon process group of their own and recycle
that much much more often than bulk of application.

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Jumpfroggy]

On Aug 10, 2:24 pm, dartdog  wrote:
> Big help!! Had just started getting using too much memory notices from
> Webfaction yesterday evening...

Yeah, I didn't even pay attention till I hit about 120-140MB on my
80MB plan and got everything turned off.  Otherwise I would have just
let it grow, not paying attention.

> The other item is using the crontab -e command from ssh to comment out
> apps you don't need to have running For some reason could not find
> that when I was looking for it and one of my sites just kept
> restarting after .stop, which I understand but did not know how to
> stop the behavior!

Good catch!  I noticed that they restart themselves, didn't bother
finding out why.

Here's my complete writeup and howto:
http://forum.webfaction.com/viewtopic.php?pid=11096

Thanks to those in this thread for helping me make this work.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[dartdog]

Big help!! Had just started getting using too much memory notices from
Webfaction yesterday evening...
The other item is using the crontab -e command from ssh to comment out
apps you don't need to have running For some reason could not find
that when I was looking for it and one of my sites just kept
restarting after .stop, which I understand but did not know how to
stop the behavior!

On Aug 10, 12:11 pm, Jumpfroggy  wrote:
> Wow, lots of good feedback here.  Thanks!
>
> On Aug 9, 6:17 pm, Graham Dumpleton 
> wrote:
>
> > Replace use of mod_python with mod_wsgi.
>
> I'm using mod_wsgi for everything already, forgot to say.  When I
> researched, it seemed like the newer/better way to do django.
>
> > Ensure you use prefork MPM
> > for Apache and delegate Django instances to run in mod_wsgi daemon
> > mode process.
> > Sorry, meant to say 'worker' rather than 'prefork'. DO NOT USE
> > PREFORK. Hmmm, that was a bad stuff up. :-)
>
> I just started using the worker MPM for one of my apps.  It didn't
> seem to give any performance or memory savings, since my app is so
> small/simple.  I'll keep an eye on it for later.
>
> Using daemon mode gives me auto-reloading by touching the .wsgi file
> (yay), but it uses more memory.  Instead of supervisor (3MB) and
> apache worker/WSGI/Django (16MB) processes, now I have supervisor
> (3MB), apache worker (3MB), and wsgi/django (15MB).  Not a huge gain,
> but just a bit more.  I'm willing to live with that for the auto-
> reloading benefit, however.
>
> > Having done that, you can use inactivity-timeout option
> > for mod_wsgi daemon processes to have the process restarted and
> > returned to low memory footprint idle process after no use for a
> > while.
>
> This is exactly what I wanted!!!  It's funny how when I finally read
> the description for something, it matches exactly what I'm looking
> for.  I just didn't know where to look.  I just tried it, and yes my
> memory baloons up to 16MB, then comes back down later.  The only
> strange thing is that I set it to 10s (very short) to test out, but it
> takes a while for the process to actually be reclaimed.  If I put
> inactivity-timeout=10, then load the page, then close the browser,
> memory usage should be reclaimed in 10 seconds, right?  It seems more
> like 1 minute wait, which is fine but strange to me.  Maybe there's
> only polling at certain intervals (checks every 1 minute), so the
> resolution isn't that fine.  Maybe it's dependent on the deadlock
> timeout?  Don't know.
>
> > Note, just ensure you don't keep loading mod_python after having
> > switched to mod_wsgi as mod_python will still consume some extra
> > memory when not in use.
>
> Yeah, never touched mod_python, so no problem here.
>
> On Aug 10, 6:44 am, Dj Gilcrease  wrote:
>
> > I just went though this process on Webfaction as well, so here is my
> > apache conf, it might help...
>
> Thanks for the post, good to see another webfaction user here.  It
> always helps to see other people's real-world configs, since examples
> usually leave out the key bits.
>
> On Aug 10, 10:25 am, Dj Gilcrease  wrote:
>
> > >> MaxRequestsPerChild 5000
> > > Don't see much point for MaxRequestsPerChild.
> > This just restarts the child after 5k requests, which helps clear out
> > some memory on occasion (I am still trying to get rrd or something run
> > on on Webfaction so I can actually map memory usage over time, and
> > start tuning Apache to my actual usage needs better)
>
> I experimented with using this as a dumb alternative to inactivity-
> timeout.  Set MaxRequestsPerChild to 1, means the WSGI process will
> restart after each HTTP request.  Very slow, but gives immediate
> memory savings.  Only viable if the MEDIA_URL is set to a separate,
> static server (which webfaction recommends and makes easy).  However,
> inactivity-timeout is still better.
>
> > >> WSGIPythonOptimize 1
> > > Am dubious that turning on optimisation helps and can cause problems
> > > if set to '2' as doc strings will be stripped.
> > I doubt it is much improvement, but I just turned it on as a test and
> > did read that 2 is a bad idea which is why I stuck with 1
>
> I'd love to see some benchmarks for this config with real apps.  The
> docs make sense... don't expect magic switches to help if you haven't
> optimized your code, so I really shouldn't even worry about this
> option yet.
>
> > >> 
> > >>     ServerName demo.digitalxero.net
> > >>     DocumentRoot /home/digitalxero/webapps/wsgi/demo
> > >>     WSGIDaemonProcess demo.digitalxero.net threads=5
> > >> maximum-requests=1 inactivity-timeout=3600
> > >> display-name=wsgi:demo_site
>
> > > That is 1 hour for inactivity timeout. They possibly want something
> > > less than that.
>
> > Ya, I picked an hour for starters and will drop it down or increase it
> > as my needs dictate (one I get something to monitory usage better 

Re: How do I minimize memory usage with WSGI and Apache?

[Jumpfroggy]

Wow, lots of good feedback here.  Thanks!

On Aug 9, 6:17 pm, Graham Dumpleton 
wrote:
> Replace use of mod_python with mod_wsgi.

I'm using mod_wsgi for everything already, forgot to say.  When I
researched, it seemed like the newer/better way to do django.

> Ensure you use prefork MPM
> for Apache and delegate Django instances to run in mod_wsgi daemon
> mode process.
> Sorry, meant to say 'worker' rather than 'prefork'. DO NOT USE
> PREFORK. Hmmm, that was a bad stuff up. :-)

I just started using the worker MPM for one of my apps.  It didn't
seem to give any performance or memory savings, since my app is so
small/simple.  I'll keep an eye on it for later.

Using daemon mode gives me auto-reloading by touching the .wsgi file
(yay), but it uses more memory.  Instead of supervisor (3MB) and
apache worker/WSGI/Django (16MB) processes, now I have supervisor
(3MB), apache worker (3MB), and wsgi/django (15MB).  Not a huge gain,
but just a bit more.  I'm willing to live with that for the auto-
reloading benefit, however.

> Having done that, you can use inactivity-timeout option
> for mod_wsgi daemon processes to have the process restarted and
> returned to low memory footprint idle process after no use for a
> while.

This is exactly what I wanted!!!  It's funny how when I finally read
the description for something, it matches exactly what I'm looking
for.  I just didn't know where to look.  I just tried it, and yes my
memory baloons up to 16MB, then comes back down later.  The only
strange thing is that I set it to 10s (very short) to test out, but it
takes a while for the process to actually be reclaimed.  If I put
inactivity-timeout=10, then load the page, then close the browser,
memory usage should be reclaimed in 10 seconds, right?  It seems more
like 1 minute wait, which is fine but strange to me.  Maybe there's
only polling at certain intervals (checks every 1 minute), so the
resolution isn't that fine.  Maybe it's dependent on the deadlock
timeout?  Don't know.

> Note, just ensure you don't keep loading mod_python after having
> switched to mod_wsgi as mod_python will still consume some extra
> memory when not in use.

Yeah, never touched mod_python, so no problem here.

On Aug 10, 6:44 am, Dj Gilcrease  wrote:
> I just went though this process on Webfaction as well, so here is my
> apache conf, it might help...

Thanks for the post, good to see another webfaction user here.  It
always helps to see other people's real-world configs, since examples
usually leave out the key bits.

On Aug 10, 10:25 am, Dj Gilcrease  wrote:
> >> MaxRequestsPerChild 5000
> > Don't see much point for MaxRequestsPerChild.
> This just restarts the child after 5k requests, which helps clear out
> some memory on occasion (I am still trying to get rrd or something run
> on on Webfaction so I can actually map memory usage over time, and
> start tuning Apache to my actual usage needs better)

I experimented with using this as a dumb alternative to inactivity-
timeout.  Set MaxRequestsPerChild to 1, means the WSGI process will
restart after each HTTP request.  Very slow, but gives immediate
memory savings.  Only viable if the MEDIA_URL is set to a separate,
static server (which webfaction recommends and makes easy).  However,
inactivity-timeout is still better.

> >> WSGIPythonOptimize 1
> > Am dubious that turning on optimisation helps and can cause problems
> > if set to '2' as doc strings will be stripped.
> I doubt it is much improvement, but I just turned it on as a test and
> did read that 2 is a bad idea which is why I stuck with 1

I'd love to see some benchmarks for this config with real apps.  The
docs make sense... don't expect magic switches to help if you haven't
optimized your code, so I really shouldn't even worry about this
option yet.

> >> 
> >> ServerName demo.digitalxero.net
> >> DocumentRoot /home/digitalxero/webapps/wsgi/demo
> >> WSGIDaemonProcess demo.digitalxero.net threads=5
> >> maximum-requests=1 inactivity-timeout=3600
> >> display-name=wsgi:demo_site
>
> > That is 1 hour for inactivity timeout. They possibly want something
> > less than that.
>
> Ya, I picked an hour for starters and will drop it down or increase it
> as my needs dictate (one I get something to monitory usage better then
> guessing)

Their default DJango installs are either mod_python or mod_wsgi.  The
mod_wsgi is embedded mode, and has no inactivity-timeout and is
StartServers 2, so by default one Django app takes 3MB + 16MB + 16MB
of memory (35MB) out of your allotted 80MB.  That means only 2.5
django apps before you run out of memory.  They have a "how to save
memory with django" howto, but it doesn't have any of these settings.
I'll try to send it updates, they'll probably appreciate it.

> > If you application does leak Python objects due to inability to break
> > reference count cycles, there shouldn't really be a need for maximum
> > 

Re: How do I minimize memory usage with WSGI and Apache?

[Dj Gilcrease]

On Mon, Aug 10, 2009 at 6:43 AM, Graham
Dumpleton wrote:
> These values for min and max threads are a bit dubious because you
> have a single process and that will have fixed 25 threads. Usually
> these are defined in multiples of ThreadsPerChild and not less like
> this. Possibly just  means the values are ignored.

The MaxClients (according to the Apache docs is #MaxServers *
#ThreadsPerChild) I just set it explicitly so I remember what it is,
though it is actually ignored


>> ThreadsPerChild 25
>> MaxRequestsPerChild 5000
>
> Don't see much point for MaxRequestsPerChild.

This just restarts the child after 5k requests, which helps clear out
some memory on occasion (I am still trying to get rrd or something run
on on Webfaction so I can actually map memory usage over time, and
start tuning Apache to my actual usage needs better)


>> WSGIPythonOptimize 1
>
> Am dubious that turning on optimisation helps and can cause problems
> if set to '2' as doc strings will be stripped.

I doubt it is much improvement, but I just turned it on as a test and
did read that 2 is a bad idea which is why I stuck with 1


>> 
>>     ServerName demo.digitalxero.net
>>     DocumentRoot /home/digitalxero/webapps/wsgi/demo
>>     WSGIDaemonProcess demo.digitalxero.net threads=5
>> maximum-requests=1 inactivity-timeout=3600
>> display-name=wsgi:demo_site
>
> That is 1 hour for inactivity timeout. They possibly want something
> less than that.

Ya, I picked an hour for starters and will drop it down or increase it
as my needs dictate (one I get something to monitory usage better then
guessing)


> If you application does leak Python objects due to inability to break
> reference count cycles, there shouldn't really be a need for maximum
> requests.

Though my app does no leak, that I have noticed anyways, I figure a
maximum-requests would be good just in case I suddenly get a flood of
traffic, it will restart the daemon even if the hour of idle time has
not passed.

I run 5 sites right now and none of them get much more then 10
visitors an hour on average, and what prompted my investigations is I
was continually sitting at 95mb of ram usage (my account is supposed
to be limited to 80) and I was using the default apache build the WF
provided, which is prefork MPM and mod_python. Now I seem to be
averaging 36mb and 55 max


> Sorry, meant to say 'worker' rather than 'prefork'. DO NOT USE
> PREFORK. Hmmm, that was a bad stuff up. :-)

ahh ok, good to know my own testing concluded corectly

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Graham Dumpleton]

On Aug 10, 8:44 pm, Dj Gilcrease  wrote:
> On Sun, Aug 9, 2009 at 4:17 PM, Graham
>
> Dumpleton wrote:
> > On Aug 10, 6:07 am, Jumpfroggy  wrote:
> >> I'm hosting a bunch of django apps on a shared host with 80MB of
> >> application memory (Webfaction in this case).  I've got a bunch of
> >> apps that are very infrequently used, but need to be online.  I've
> >> already changed the httpd.conf:
>
> I just went though this process on Webfaction as well, so here is my
> apache conf, it might help
>
> LoadModule rewrite_module modules/mod_rewrite.so
> LoadModule wsgi_module modules/mod_wsgi.so
> LoadModule deflate_module modules/mod_deflate.so
> KeepAlive Off
> Timeout 300
> ServerLimit 1
> StartServers 1
> MaxClients 25
> MinSpareThreads 2
> MaxSpareThreads 4

These values for min and max threads are a bit dubious because you
have a single process and that will have fixed 25 threads. Usually
these are defined in multiples of ThreadsPerChild and not less like
this. Possibly just  means the values are ignored.

> ThreadsPerChild 25
> MaxRequestsPerChild 5000

Don't see much point for MaxRequestsPerChild.

> WSGIPythonOptimize 1

Am dubious that turning on optimisation helps and can cause problems
if set to '2' as doc strings will be stripped.

> 
>     ServerName demo.digitalxero.net
>     DocumentRoot /home/digitalxero/webapps/wsgi/demo
>     WSGIDaemonProcess demo.digitalxero.net threads=5
> maximum-requests=1 inactivity-timeout=3600
> display-name=wsgi:demo_site

That is 1 hour for inactivity timeout. They possibly want something
less than that.

If you application does leak Python objects due to inability to break
reference count cycles, there shouldn't really be a need for maximum
requests.

>     WSGIScriptAlias / /home/digitalxero/webapps/wsgi/demo/django.wsgi
>     ErrorLog /home/digitalxero/webapps/wsgi/demo/error.log
> 
>
> ...
>
> > Replace use of mod_python with mod_wsgi. Ensure you use prefork MPM
>
> Why prefork, in my testing I found that the worker MPM used less
> memory in the long run, it uses a little bit more during idle time,
> but scales better

Sorry, meant to say 'worker' rather than 'prefork'. DO NOT USE
PREFORK. Hmmm, that was a bad stuff up. :-)

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Dj Gilcrease]

On Sun, Aug 9, 2009 at 4:17 PM, Graham
Dumpleton wrote:
> On Aug 10, 6:07 am, Jumpfroggy  wrote:
>> I'm hosting a bunch of django apps on a shared host with 80MB of
>> application memory (Webfaction in this case).  I've got a bunch of
>> apps that are very infrequently used, but need to be online.  I've
>> already changed the httpd.conf:

I just went though this process on Webfaction as well, so here is my
apache conf, it might help

LoadModule rewrite_module modules/mod_rewrite.so
LoadModule wsgi_module modules/mod_wsgi.so
LoadModule deflate_module modules/mod_deflate.so
KeepAlive Off
Timeout 300
ServerLimit 1
StartServers 1
MaxClients 25
MinSpareThreads 2
MaxSpareThreads 4
ThreadsPerChild 25
MaxRequestsPerChild 5000

WSGIPythonOptimize 1


ServerName demo.digitalxero.net
DocumentRoot /home/digitalxero/webapps/wsgi/demo
WSGIDaemonProcess demo.digitalxero.net threads=5
maximum-requests=1 inactivity-timeout=3600
display-name=wsgi:demo_site
WSGIScriptAlias / /home/digitalxero/webapps/wsgi/demo/django.wsgi
ErrorLog /home/digitalxero/webapps/wsgi/demo/error.log


...

> Replace use of mod_python with mod_wsgi. Ensure you use prefork MPM

Why prefork, in my testing I found that the worker MPM used less
memory in the long run, it uses a little bit more during idle time,
but scales better

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: How do I minimize memory usage with WSGI and Apache?

[Graham Dumpleton]

On Aug 10, 6:07 am, Jumpfroggy  wrote:
> I'm hosting a bunch of django apps on a shared host with 80MB of
> application memory (Webfaction in this case).  I've got a bunch of
> apps that are very infrequently used, but need to be online.  I've
> already changed the httpd.conf:
>     ServerLimit 1
> Instead of the default "ServerLimit 2".  With the default, each app
> spawns 3 apache processes; one manager, 2 worker processes.  With
> ServerLimit 1, there is only a manager and a single worker.  This cuts
> down on a lot for low-traffic sites.
>
> However, I'd like to see what the bare minimum memory usage can be.
> I've made sure DEBUG = False.  I also separated the static media to a
> separate server with MEDIA_ROOT and MEDIA_URL.  When I first start
> apache, the manager and worker processes use about 3MB (RSS) each.
> After the first access with a browser, the manager stays the same but
> the worker uses 16MB and stays there until apache is restarted.
>
> Ideally, I'd love it if the memory usage go back to the initial state
> (3MB) after a period of inactivity.  I'm curious; what is being stored
> in memory?  Are there other things I can do to reduce memory usage?
>
> The one thing I've seen is the "MaxRequestsPerChild" apache setting.
> As far as I can tell, it'll recreate the worker process after X
> requests.  But this seems like it might adversely affect performance,
> since I need something time-based, and not # requests based.  With
> MaxRequestsPerChild, I'm guessing if I set it to 10 but only had 9
> requests, it would sit there using up memory until the tenth request,
> which is not ideal.  With MaxRequestsPerChild set to 1, I'd be
> guaranteed low memory usage but would also recreate the worker process
> for each request, which seems like a bad idea.
>
> My goal is to have many apps running at bare-minimum memory usage, and
> only the frequently accessed sites taking up ~16MB memory each.  And
> when the active sites are dormant (late at night, etc), I'd like those
> to lower their memory use as well.  Is there any way to do this?
>
> Thanks! (and apologies if I've missed some obvious documentation...
> I've gone through a few guides, but nothing seems to have helped).

Replace use of mod_python with mod_wsgi. Ensure you use prefork MPM
for Apache and delegate Django instances to run in mod_wsgi daemon
mode process. Having done that, you can use inactivity-timeout option
for mod_wsgi daemon processes to have the process restarted and
returned to low memory footprint idle process after no use for a
while.

Note, just ensure you don't keep loading mod_python after having
switched to mod_wsgi as mod_python will still consume some extra
memory when not in use.

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---