Re: better uptime with a fallback app

[Tom Evans]

On Wed, 2009-07-22 at 04:47 -0700, Graham Dumpleton wrote:
> 
> 
> On Jul 22, 9:34 pm, Le Roux Bodenstein  wrote:
> > > If it is to bring down application for maintenance, seems like it
> > > would be easier to use Apache/mod_wsgi in daemon mode.
> >
> > I'll give mod_wsgi a go. To be honest I never looked at it before
> > simply because it is tied to apache. But if it can easily do
> > everything I need using a reasonable amount of memory and provide
> > decent performance, then I'll allow it ;)
> >
> > My app pre-dates mod_wsgi and at the time mod_python just seemed too
> > bloated.
> 
> There is a lot of myth to that. Although older versions of mod_python
> had some memory leaks, that has in the main been addressed. The
> problems with memory bloat were Python installations with no shared
> library for Python and bad configuration of Apache. For a bit of a
> discussion see:
> 
>   http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
> 
> > I've never been an apache fan and memory usage is/was my main
> > concern which is why I leaned towards lighttpd. At the time I felt
> > that it performed better with serving up static files, but I haven't
> > actually benchmarked it in ages. I suppose you can run lighttpd/nginx
> > in front of apache,
> 
> And I would also suggest you run nginx in front. It is good at static
> file serving and also because of the way it handles proxying, isolates
> Apache from slow clients. This means that Apache is being used for as
> little time as possible as nginx will act as a buffer and handle the
> talking to the slow client. You therefore get optimal use of Apache
> resources, much better than if Apache was the front facing web server
> and was handling both static and dynamic requests. Behind nginx,
> Apache doesn't have to worry about keep alive either.
> 
> > but all apache is really doing by then is act as a
> > proxy that converts http into something mod_wsgi daemons understand
> > which just feels terribly bloated to me..
> 
> How much memory a Python application takes up is not really going to
> vary whatever mechanism you use. So, if you run a multiprocess
> configuration with 5 process, be it with mod_wsgi daemon mode, flup or
> mod_proxy load balanced, that is still 5 times whatever amount of
> memory your application requires. There is nothing about Apache that
> suddenly means your application needs twice as much memory or
> otherwise bloats out more than it would with other mechanisms. There
> is a lot of FUD and misunderstanding around using Python with Apache,
> it is not as evil as some would like to portray, you just need to
> configure it properly and understand how it works to make best use of
> it.
> 
> Overall, the only difference should be the minimal amount of memory
> Apache itself adds to each process, but in the context of your typical
> fat Python web application that is small enough to be neglible.
> 
> Graham

I probably wouldn't have replied to this, but it riled me with the
implication that apache cannot serve dynamic content and static content
efficiently (maybe you didn't mean that, and I'm getting the wrong end
of the stick, this happens to be my pet peeve!). Just as there is plenty
of FUD around using python with apache, there is just as much
surrounding apache, with people suggesting nginx, lighty etc as
'solutions'.

Our solution is simple; configure apache correctly for your needs, and
it behaves correctly in all scenarios.

First off, we use the event MPM for apache. This is a threaded MPM that
uses event queues for various parts of the request processing, and is
extremely efficient. 
Secondly, and just as important, read your config file, understand each
directive and decide if YOU need it on YOUR site. If you don't use
apache's content negotiation, then don't load mod_negotation. 

The downside of the event MPM is that it is threaded, so throw out any
mod_ modules, even if the core is thread safe, it only takes one
extension that is not to bork your server. For example, django is
purported to be thread safe, but there are many python libraries that
are not.

Instead, we run each dynamic application separately, as a fastcgi or
wsgi application. This gives you the benefit of tuning the number of
worker processes by application, and allows you to use your OS to
control resources for the application. 
We run all of ours as fastcgi instances, as that is language neutral,
and allows us to run any webapp in the same manner.

Since we are running the event MPM, apache scales efficiently to handle
large volumes of requests. 
Apache directly serves any static content, and it is efficient to use an
apache worker thread to do this, rather than insert something like nginx
in front of it. 
Dynamic requests are handled by mod_fastcgi, and (in our usual setup)
the apps that actually end up handling the request run on different
boxes.

All in all, our single httpd server handles between 3-8 million requests

Re: better uptime with a fallback app

[Le Roux Bodenstein]

> The only criticism I have seen of lighttpd is that is has been a long
> time since last release and that it can leak memory. Don't know how
> valid that is, so I could be generating my own FUD here. :-)

Well. I was running it for more than a year without a restart at some
stage and I didn't notice any memory leaks. Since then I upgraded to a
newer version (because some modules I wanted to use just wouldn't work
for me, I added some extra ssl certificates, ip addresses and things)
and that's been running for weeks with no increase in memory usage, so
I'm pretty sure that if there are memory leaks it doesn't affect me.

I've been meaning to give nginx a go, but learning a new web server's
configuration language and tuning the config can take up a lot of
time. And why replace something that's not broken? I suspect it is the
darling at the moment because someone somewhere ran a benchmark on
some pointless do nothing / "hello world" page/file/app and squeezed
out a few more requests per second. Which (as you have stated
repeatedly elsewhere) is not going to make any noticeable difference
to a big, fat, slow python web app's performance.

Also, I don't think many people even heard of nginx a few years ago
when I made the decision to go with lighttpd..
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: better uptime with a fallback app

[Graham Dumpleton]

On Jul 22, 10:17 pm, Le Roux Bodenstein  wrote:
> >  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa...
>
> Wow. Thanks. I'm reading your entire blog now - it is really helpful.
> I just realised how little I know about this. Your blog is certainly
> the best resource on this topic I've ever seen.
>
> Since I'm currently running lighttpd which has a fair bit of config
> that I don't want to move, I'm going to probably run that in front of
> apache rather than nginx. (at least for now and assuming it "just
> works")

The only criticism I have seen of lighttpd is that is has been a long
time since last release and that it can leak memory. Don't know how
valid that is, so I could be generating my own FUD here. :-)

Certainly, nginx seems to be the darling at the moment and not sure
lighttpd has same properties that nginx has as far as isolating Apache
from slow clients. For nginx, this is in part the result of nginx
buffering request content before proxying on to back end server.

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: better uptime with a fallback app

[Le Roux Bodenstein]

>  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa...

Wow. Thanks. I'm reading your entire blog now - it is really helpful.
I just realised how little I know about this. Your blog is certainly
the best resource on this topic I've ever seen.

Since I'm currently running lighttpd which has a fair bit of config
that I don't want to move, I'm going to probably run that in front of
apache rather than nginx. (at least for now and assuming it "just
works")

Le Roux


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: better uptime with a fallback app

[Graham Dumpleton]

On Jul 22, 9:34 pm, Le Roux Bodenstein  wrote:
> > If it is to bring down application for maintenance, seems like it
> > would be easier to use Apache/mod_wsgi in daemon mode.
>
> I'll give mod_wsgi a go. To be honest I never looked at it before
> simply because it is tied to apache. But if it can easily do
> everything I need using a reasonable amount of memory and provide
> decent performance, then I'll allow it ;)
>
> My app pre-dates mod_wsgi and at the time mod_python just seemed too
> bloated.

There is a lot of myth to that. Although older versions of mod_python
had some memory leaks, that has in the main been addressed. The
problems with memory bloat were Python installations with no shared
library for Python and bad configuration of Apache. For a bit of a
discussion see:

  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

> I've never been an apache fan and memory usage is/was my main
> concern which is why I leaned towards lighttpd. At the time I felt
> that it performed better with serving up static files, but I haven't
> actually benchmarked it in ages. I suppose you can run lighttpd/nginx
> in front of apache,

And I would also suggest you run nginx in front. It is good at static
file serving and also because of the way it handles proxying, isolates
Apache from slow clients. This means that Apache is being used for as
little time as possible as nginx will act as a buffer and handle the
talking to the slow client. You therefore get optimal use of Apache
resources, much better than if Apache was the front facing web server
and was handling both static and dynamic requests. Behind nginx,
Apache doesn't have to worry about keep alive either.

> but all apache is really doing by then is act as a
> proxy that converts http into something mod_wsgi daemons understand
> which just feels terribly bloated to me..

How much memory a Python application takes up is not really going to
vary whatever mechanism you use. So, if you run a multiprocess
configuration with 5 process, be it with mod_wsgi daemon mode, flup or
mod_proxy load balanced, that is still 5 times whatever amount of
memory your application requires. There is nothing about Apache that
suddenly means your application needs twice as much memory or
otherwise bloats out more than it would with other mechanisms. There
is a lot of FUD and misunderstanding around using Python with Apache,
it is not as evil as some would like to portray, you just need to
configure it properly and understand how it works to make best use of
it.

Overall, the only difference should be the minimal amount of memory
Apache itself adds to each process, but in the context of your typical
fat Python web application that is small enough to be neglible.

Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: better uptime with a fallback app

[Le Roux Bodenstein]

> If it is to bring down application for maintenance, seems like it
> would be easier to use Apache/mod_wsgi in daemon mode.

I'll give mod_wsgi a go. To be honest I never looked at it before
simply because it is tied to apache. But if it can easily do
everything I need using a reasonable amount of memory and provide
decent performance, then I'll allow it ;)

My app pre-dates mod_wsgi and at the time mod_python just seemed too
bloated. I've never been an apache fan and memory usage is/was my main
concern which is why I leaned towards lighttpd. At the time I felt
that it performed better with serving up static files, but I haven't
actually benchmarked it in ages. I suppose you can run lighttpd/nginx
in front of apache, but all apache is really doing by then is act as a
proxy that converts http into something mod_wsgi daemons understand
which just feels terribly bloated to me..
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: better uptime with a fallback app

[Graham Dumpleton]

On Jul 22, 8:19 pm, Le Roux Bodenstein  wrote:
> I'm trying to keep my django powered cms online even when I have to
> restart the app or take it down for maintenance.
>
> My rendered pages are generally quite cacheable and they only change
> if someone content manages a page or someone comments, etc. which
> happens relatively rarely.
>
> I have a very crude page caching mechanism where I basically save the
> html along with the last-modified date and the full url in a table in
> the database (or wherever). Whenever anything on the site changes, I
> just set a site-wide last-modified datetime. Then whenever a page gets
> loaded, I check the last modified date/time of the cached page against
> the site-wide one and either return the cached copy or I re-render it,
> save the new one and return it. This almost always happens when the
> admin/content manager/author person checks the page he just edited, so
> by the time a "normal" user comes along the new page is already in the
> cache. This already gets me a near-100% cache-hit rate.
>
> (It is actually slightly more complicated than this, but that is not
> important because most of the logic has to do with figuring out when
> it is safe to write something to the cache.)
>
> So basically.. It is very easy to write a tiny/dumb little wsgi app
> that can just check the cached pages table to see if there is a cached
> page for a specific url, skip the freshness check and just return
> that. I want to use this little app as an automatic fallback in case
> the main one is down for maintenance. It won't be able to keep the /
> admin/ interface up, but in that case it can just return a "Don't
> worry. Your site is still up and running, but you won't be able to
> manage your site for the next 5 minutes while we do maintenance." type
> error page.
>
> Anyway.. obviously I need to be running a proxy that will check the
> main app first and if that's down (nothing running on that port) it
> should fall back to the backup backend. I think it should be able to
> do this immediately without any health monitoring that might take up
> to seconds to realise that the backend is gone, then seconds to
> realise that it is up again resulting in errors in the meantime.
>
> I tried Varnish, but couldn't really get it to do what I want and I
> also had unrelated intermittent errors that I just couldn't debug/fix
> and in the end I realised I really don't need a caching proxy at this
> stage. I looked through the documentation and features lists of
> various other reverse proxies and I couldn't find anything that
> supports this straight out of the box. I'm on the verge of writing my
> own, but thinking that there must be something out there already.
>
> My setup is currently:
>
> the internet -> lighttpd -> django app via fastcgi (flup)
>
> I'm thinking of moving to:
>
> the internet -> lighttpd -> some proxy ->
> (1) pure python wsgi/web server or
> (2) backup pure python little webserver
> (so the proxy tries 1, then 2)
>
> In all these cases lighttpd handles all the static file requests and
> then proxies the dynamic ones along (either via plain http or
> fastcgi). I'm not so worried about keeping lighttpd up and running at
> this stage - it is very stable. I'm more concerned about being able to
> gracefully take the app down.
>
> I'm looking at switching the flup based process over to 
> spawning:http://pypi.python.org/pypi/Spawning/which sounds like it could keep
> my app up and running during normal/routine restarts, but it won't
> help with slightly longer (+/- 5 minute) maintenance periods where I
> have to take the app down to be safe.
>
> Does anyone have any suggestions? How do other people handle this?

If it is to bring down application for maintenance, seems like it
would be easier to use Apache/mod_wsgi in daemon mode.

All you would need to do is swap out the WSGI script file which is the
entry point with your mini WSGI application to be used during
maintenance work. When the file is swapped the modification time
changes and so mod_wsgi will automatically restart the application
processes and swap to the maintenance mode application. When done,
just swap the WSGI script file back again and it will restart again
automatically.

One could even get more complicated than this and have the two WSGI
applications always running but in distinct daemon process groups.
Using a bit of dynamic configuration with mod_wsgi, you could without
even needing to restart anything, just redirect the traffic to the
daemon process group running the maintenance mode application. You
could even allow just traffic from specific clients to still go
through to the original application if the maintainers needed to
interact with the application even when maintenance is being done.
Still having access means you can make changes and trigger a restart
of the application and then test it at its real location before you
start allowing all traffic to starting hitting it again.