2008/10/21 henrybaxter <[EMAIL PROTECTED]>: > > Yes I am using Django > >> Can you explain better what happens for that 1 request in 5000? >> > > I get a 404 notification via Django's admin 404/500 error email > service that a URL is broken.
The random 404/500 responses may be accounted for by other things. A few things come to mind. The first is that you have turned on extended multi language error documents, or otherwise have defined ErrorDocument directives. It is typical to map these to pages in document root, but if Django is mounted at the root URL, a error in something outside of django, such as a request against a non existent static resource, erroneously from application, or by SPAM BOT, can result in Apache sending a request for a 404 error response page to Django. Because Django will not know about it, it would in turn return a 404 error response, which in turn causes Apache to return a 500 error response as it would be expecting a 200 okay response. Because the request for the error document is a sub request, it doesn't show as a request in the Apache error logs and with Django not doing any logging of requests either, it wouldn't be evident anywhere what is going on. You would only know by looking in the access logs and noting that the URL wasn't a valid one to begin with. In your case, since you say the URL is valid and works otherwise, this explaination possibly doesn't account for the problem as request as seen within Django and possibly sent in email would be for error document URL. The next possibility is that Django itself is randomly raising some sort of error in its response handling chain which is outside of the context of its catch all exception handling and error page generation. Django 1.0 at least is supposed to be thread safe, except for some uncommon corner cases, but older versions of Django do potentially have some issues with multithreading. Since though you were using multithreading with multi process configuration and weren't seeing the issue, multithreading problems would again also not seem likely. Although it again relies on Django not catching exceptions properly and generating its own error page, another problem can be where a client connection is closed prematurely, resulting in a read against request content, or sending of response generating an IOError exception. Often handler code doesn't deal with this and it propagates back upwards. Since Django shouldn't be allowing such unexpected errors to propagate back up, that also doesn't make sense. Anyway, do you have an example of the email that Django sends you so can see what it describes. Scrub/change any sensitive information as needs be. > However, that URL is 100% fine when I > try it myself. I realized after tons of debugging output that the > worker server for example1.com was using at least 2 things from > example2.com's settings: a custom setting in settings.py SITE_NAME = > 'example2.com' and also example2.com's database connection. I output > all the request['META'] information and sure enough everything lined > up with the server thinking this was an example2.com request, but it > was actually to the example1.com virtual host. Did you by chance dump out what mod_wsgi.process_group and mod_wsgi.application_group were from the WSGI environment? Dumping out those along with value of DJANGO_SETTINGS_MODULE from os.environ along with other stuff from WSGI environment such as REQUEST_URI and SERVER_NAME. Not sure if some of the debugging WSGI middleware described in: http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response could be of use. It may be possible to adapt the first example to remember stuff and dump out debug files when 404/500 errors are seen when not expected. This may give you better information about the request and what mod_wsgi thought it was being asked to do. >> I'll explain better later when have a chance, but there are issues >> with Python and multiple sub interpreters in respect of environment >> variable leakage, but if the only os.environ variable being set was >> DJANGO_SETTINGS_MODULE, it wouldn't usually present as a problem. >> >> That said, I have heard a couple of times of strange mixing still >> occurring whereby using distinct daemon processes for each instance >> solved the problem. > > I used to have separate DaemonProcessGroup lines for each virtual > host. I decided to merge them (since I'd done it in testing and it > worked fine) to allow processes to be shared (I figured this would > provide better ability to absorb traffic spikes on an individual > website, that kind of thing). At that time there were no problems. I > am fairly certain that merging them caused the problem, but my logging > at that point was not good enough so I cannot get a precise time to > correlate with the change in settings. > > Furthermore, I have since added WSGIApplicationGroup %{SERVER} to each > virtual host, and it has magically stopped all the problems. After > 100,000+ requests there have been zero problems. > > ... > > As I mentioned above the WSGIApplicationGroup %{SERVER} setting seems > to have cured everything, so what I'm curious to know is, what did > that change? Isn't the default setting almost identical? Overall, if default WSGIApplicationGroup setting of %{RESOURCE} is being applied, they should be separated. As I said, have seen odd cases, mainly with mod_python though, where sub interpreter separation seemed to be failing. In mod_python, the problems in some cases were somehow linked to use of ServerAlias directive in Apache, but if you are proxying where server name would always be the same, and with no indication that you have used ServerAlias anyway, this wouldn't come into it. I don't quite see how changing it to %{SERVER} would make a difference. To explain the difference, for %{RESOURCE} would expect to see in WSGI environment: mod_wsgi.application_group: 'dangermouse.isd.rta.nsw.gov.au:8224|/wsgi/scripts/echo.py' The value on right side is the name given to the sub interpreter. Thus qualification is by server name, server port and the mount point of the WSGI script file. In your case would expect these to be: example1.com:8000| example2.com:8001| >From memory the mount point will be empty when at root as it takes on value of SCRIPT_NAME which is empty for root, with PATH_INFO actually being '/'. Memory could be faulty here though. By setting it to %{SERVER} would instead just be: example1.com:8000 example2.com:8001 Depending on Apache configuration, guess they might be IP addresses instead, but reasonably sure it should take on value of ServerName directive. The C code which does this isn't too complicated: if (!strcmp(name, "{RESOURCE}")) { h = r->server->server_hostname; p = ap_get_server_port(r); n = wsgi_script_name(r); if (p != DEFAULT_HTTP_PORT && p != DEFAULT_HTTPS_PORT) return apr_psprintf(r->pool, "%s:%u|%s", h, p, n); else return apr_psprintf(r->pool, "%s|%s", h, n); } if (!strcmp(name, "{SERVER}")) { h = r->server->server_hostname; p = ap_get_server_port(r); if (p != DEFAULT_HTTP_PORT && p != DEFAULT_HTTPS_PORT) return apr_psprintf(r->pool, "%s:%u", h, p); else return h; } > If this will help, there are actually 4 total websites spread across 4 > worker servers (these are meant to spread the load and at the same > time provide redundancy, that's why there isn't one server per > website) and then two load balancers using heartbeat to enhance uptime > providing load balancing and also any SSL processing (which is fairly > small at the moment). Finally there is a separate dedicated database > server group. They all use the same group, but a different database > instance each. All software involved is from the debian lenny > distribution, and Django latest svn (except for a one line patch on > the contrib.admin code). One of the websites gets at least 60,000 > visitors per day, another 30,000, another 5,000, and the final one > almost zero. All four websites have identical settings/wsgi scripts/ > etc as posted and discussed here, it just seemed redundant to include > them. I can guarantee they are all appropriately different though, > they are automatically generated and checked very thoroughly. Even if in separate sub interpreters of same process, each Django instance will take up same amount of memory as if in different processes. Thus not sure what you think is gained from running them in same process. You did say 'provide better ability to absorb traffic spikes on an individual website', but am not sure how. If anything by putting them in all one process group, you are restricting the number of concurrent requests across all sites to 15*6 whereas if separate daemon process groups, each site could have maximum of 15*6 concurrent requests. Just to make sure there is no risk of problems from environment pollution, or with third party C extension modules are sub interpreters, I would have delegated each site to own daemon process group and also force them to run in main Python interpreter in process since only instance in the process. Thus: Listen 192.168.0.14:8000 <VirtualHost 192.168.0.14:8000> ServerName example1.com WSGIDaemonProcess example1.com user=www-data group=www-data processes=6 threads=15 python-path=/var/code/:/var/code/drat/ WSGIScriptAlias / /var/code/drat/settings/example1/live.wsgi WSGIProcessGroup example.com WSGIApplicationGroup %{GLOBAL} SetEnvIf X_FORWARDED_PROTO https HTTPS 1 </VirtualHost> # example2.com Listen 192.168.0.14:8001 <VirtualHost 192.168.0.14:8001> ServerName example2.com WSGIDaemonProcess example2.com user=www-data group=www-data processes=6 threads=15 python-path=/var/code/:/var/code/drat/ WSGIScriptAlias / /var/code/drat/settings/example2/live.wsgi WSGIProcessGroup example2.com WSGIApplicationGroup %{GLOBAL} SetEnvIf X_FORWARDED_PROTO https HTTPS 1 </VirtualHost> For the less used sites, you could cut down on number of processes and/or threads and even perhaps enable inactivity-timeout. This would allow you to drop down overall memory usage on the machine. Use of separate processes allows you to also set maximum-requests individually if there is a need to recycle processes at different rates due to memory creap you can't eliminate. Graham --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en -~----------~----~----~----~------~----~------~--~---
