> > Yes I am using Django > > >> Can you explain better what happens for that 1 request in 5000? > > > I get a 404 notification via Django's admin 404/500 error email > > service that a URL is broken. > > The random 404/500 responses may be accounted for by other things. A > few things come to mind.
> The first is that you have turned on extended multi language error > documents, or otherwise have defined ErrorDocument directives. This is not the case > request against a non existent static resource, erroneously from > application, or by SPAM BOT, can result in Apache sending a request > for a 404 error response page to Django. I'm extremely familiar with these :) But they have been ruled out. > ...You would only know by looking in the access logs > and noting that the URL wasn't a valid one to begin with. This is very interesting, and I didn't know how that worked exactly. However, this can also be ruled out from the extra debugging output and further examination of the logs. > Anyway, do you have an example of the email that Django sends you so > can see what it describes. Scrub/change any sensitive information as > needs be. It just provides the very normal 404 error email with the referrer and the URL requested. It is just what peaked my interest, but once I was interested I just used them to gauge how big a problem it was (how many emails I got) but not for actually learning more about the problem. For that I examined the apache error logs on the balancer and each worker, and when that was still confusing, I started outputting a lot more debugging information. > Did you by chance dump out what mod_wsgi.process_group and > mod_wsgi.application_group were from the WSGI environment? Dumping out > those along with value of DJANGO_SETTINGS_MODULE from os.environ along > with other stuff from WSGI environment such as REQUEST_URI and > SERVER_NAME. I did, all of the above. That was my first clue for what the heck was going on. However after resolving the issue as I described in my last post/email, I cleaned the logs and waited for new information. Since the solution fixed the problem, I don't think I actually have a record of what was output. I'm as frustrated as you! What I can tell you from memory though is that all of the readouts showed that the server thought it was example1.com, but the REQUEST_URI indicated clearly that the traffic was for example2.com. I just wish I could remember precisely what mod_wsgi.application_group said. I'm tempted to make the problem happen again on our traffic just to get to the bottom of this! I will let you know if I do so. > Not sure if some of the debugging WSGI middleware described in: > > http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Re... > > could be of use. That looks really useful, mod_wsgi is pretty cool! As mentioned, without a problem, more debugging output is currently not useful. I will definitely use it as appropriate the next time I post a problem to this mailing list. > >> I'll explain better later when have a chance, but there are issues > >> with Python and multiple sub interpreters in respect of environment > >> variable leakage, but if the only os.environ variable being set was > >> DJANGO_SETTINGS_MODULE, it wouldn't usually present as a problem. I'm quite certain that was the only environment variable being set. > > >> That said, I have heard a couple of times of strange mixing still > >> occurring whereby using distinct daemon processes for each instance > >> solved the problem. This was my original situation before I 'merged' into one daemon process group, and the problem did not occur then. > > As I said, have seen odd cases, mainly with mod_python though, where > sub interpreter separation seemed to be failing. In mod_python, the > problems in some cases were somehow linked to use of ServerAlias > directive in Apache, but if you are proxying where server name would > always be the same, and with no indication that you have used > ServerAlias anyway, this wouldn't come into it. Nope no ServerAlias. > > I don't quite see how changing it to %{SERVER} would make a > difference. To explain the difference, for %{RESOURCE} would expect to > see in WSGI environment: > > mod_wsgi.application_group: > 'dangermouse.isd.rta.nsw.gov.au:8224|/wsgi/scripts/echo.py' > > The value on right side is the name given to the sub interpreter. Thus > qualification is by server name, server port and the mount point of > the WSGI script file. In your case would expect these to be: > > example1.com:8000| > example2.com:8001| > > From memory the mount point will be empty when at root as it takes on > value of SCRIPT_NAME which is empty for root, with PATH_INFO actually > being '/'. Memory could be faulty here though. > > By setting it to %{SERVER} would instead just be: > > example1.com:8000 > example2.com:8001 > > Depending on Apache configuration, guess they might be IP addresses > instead, but reasonably sure it should take on value of ServerName > directive. > > The C code which does this isn't too complicated: > > if (!strcmp(name, "{RESOURCE}")) { > h = r->server->server_hostname; > p = ap_get_server_port(r); > n = wsgi_script_name(r); > > if (p != DEFAULT_HTTP_PORT && p != DEFAULT_HTTPS_PORT) > return apr_psprintf(r->pool, "%s:%u|%s", h, p, n); > else > return apr_psprintf(r->pool, "%s|%s", h, n); > } > > if (!strcmp(name, "{SERVER}")) { > h = r->server->server_hostname; > p = ap_get_server_port(r); > > if (p != DEFAULT_HTTP_PORT && p != DEFAULT_HTTPS_PORT) > return apr_psprintf(r->pool, "%s:%u", h, p); > else > return h; > } > This is exactly what I thought was/should be happening, so I wonder if it is at all possible that the default settings of %{RESOURCE} was somehow changed, either in my distribution (I'm using debian lenny, and the mod-wsgi packaged by them) or in my own configuration (extremely unlikely, but of course I could have missed something). I'm really stumped on how the change could have had any impact. Apart from only making the change once (so not repeating the 'experiment'), nothing else that I can think of could be responsible for the change. Since I cannot provide any more useful information on the problem (because I'm not working with the problem at the moment) I respectfully suggest that we end this line of discussion, and only resurrect it if A) the problem happens again on its own, or B) I gather the courage to change it back to the bad configuration to further study the issue with the debugging tips you have provided. > Even if in separate sub interpreters of same process, each Django > instance will take up same amount of memory as if in different > processes. Thus not sure what you think is gained from running them in > same process. That makes sense, and I guess I didn't think about that. > You did say 'provide better ability to absorb traffic > spikes on an individual website', but am not sure how. If anything by > putting them in all one process group, you are restricting the number > of concurrent requests across all sites to 15*6 whereas if separate > daemon process groups, each site could have maximum of 15*6 concurrent > requests. This may be rooted in a fundamental understanding of mine, but here is my logic: I wanted to keep the total number of processes and threads down while allowing each site to use as much of the machine's resources as needed in a high traffic situation. I started out with 2 processes * 15 threads in each site, so at the time 6 processes * 15 threads of 'memory usage and machine usage' in my mind. But (I thought) the heaviest hit site was limited to 2 * 15 = 30 concurrent requests. If I was to move all the websites to 6 * 15 threads, then the 'memory usage and machine usage' would go to a maximum of 18 * 15 (actually with four servers now 24 * 15) for the ability of a single site to handle 6 * 15 simultaneous requests. By using 6 * 15 across all sites together, if one site required zero requests and one site required 30 requests, the same ability could be provided *but* the total number of requests for the machine (and therefore cpu/machine usage) was still limited. I'm beginning to wonder if I'm totally wrong in this understanding, but there it is! > I would have delegated each site to own daemon process > group and also force them to run in main Python interpreter in process > since only instance in the process. Thus: > > Listen 192.168.0.14:8000 > <VirtualHost 192.168.0.14:8000> > ServerName example1.com > WSGIDaemonProcess example1.com user=www-data group=www-data processes=6 > threads=15 python-path=/var/code/:/var/code/drat/ > WSGIScriptAlias / /var/code/drat/settings/example1/live.wsgi > WSGIProcessGroup example.com > WSGIApplicationGroup %{GLOBAL} > SetEnvIf X_FORWARDED_PROTO https HTTPS 1 > </VirtualHost> > > # example2.com > Listen 192.168.0.14:8001 > <VirtualHost 192.168.0.14:8001> > ServerName example2.com > WSGIDaemonProcess example2.com user=www-data group=www-data processes=6 > threads=15 python-path=/var/code/:/var/code/drat/ > WSGIScriptAlias / /var/code/drat/settings/example2/live.wsgi > WSGIProcessGroup example2.com > WSGIApplicationGroup %{GLOBAL} > SetEnvIf X_FORWARDED_PROTO https HTTPS 1 > </VirtualHost> > > For the less used sites, you could cut down on number of processes > and/or threads and even perhaps enable inactivity-timeout. This would > allow you to drop down overall memory usage on the machine. Use of > separate processes allows you to also set maximum-requests > individually if there is a need to recycle processes at different > rates due to memory creap you can't eliminate. I believe I do understand the logic behind your suggestion, and I think it makes a lot of sense. Since my current situation should (by my understanding) provide maximum efficiency and does also seem to work quite well with the application that I'm running, I'm going to stick with it for now. If I get the impression there are any further problems at all, I will first debug very carefully using the tools you have shown me, and then switch to the above configuration to hopefully remove any possible problem. Thanks for your awesome and detailed responses! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en -~----------~----~----~----~------~----~------~--~---
