2010/1/11 qrilka <[email protected]>: > Thanks for your reply, Graham. > It was quite clear before that I need to spend more time on proper > server configuration (e.g. Apache now isn't quite tuned at least I > need to separate python from PHP). I will look into you proposal on > using nginx for proxying long-running requests. > BTW I removed maximum-requests but still see 'Premature end of script > headers' messages in error log. > What could be a reason for that? > Are there any other restarts for mod_wsgi except changed wsgi file and > maximum-requests option?
Could also occur if someone is doing an: apachectl graceful Ie., Apache graceful restart. Can occur here because Apache will allow its own server child process to keep running until active requests complete, but due to way APR library handles the other processes such as mod_wsgi daemon processes, it will kill them off regardless after 3 seconds. Thus, mod_wsgi daemon process goes away and the Apache server child process proxying request to the daemon process will then see connection close and get that error message. Presumably an Apache graceful shutdown could cause something similar. The question is therefore if anyone is doing graceful restarts on Apache at same time as you still see the message. The only other times that error message could arise is if the mod_wsgi daemon process crashed, or if someone has sent an explicit signal to the mod_wsgi daemon process to make it shutdown. Graham > Many thanks for you help. > > Best regards, > Kirill Zaborsky > > On Jan 11, 3:23 am, Graham Dumpleton <[email protected]> > wrote: >> 2010/1/8 qrilka <[email protected]>: >> >> > OK, I'll investigate how that could be solved. >> > Actually application is very simple and the only problem could be with >> > some operations on large images. >> > But I do not see how that could lead to some operations taking more >> > than 1 second. >> >> That time of 5 seconds isn't just for processing of the image but also >> inclusive of the time it takes to upload the image to the application >> if what you are doing first involves an upload. >> >> Thus, if dealing with large images or slow HTTP clients and clients >> are talking direct to Apache, then you may well exceed that time. >> >> What you can do to partly isolate yourself from problem of slow HTTP >> clients is to put nginx proxy in front of Apache. At least for files >> up to some default, nginx will buffer the upload before actually >> triggering the proxy to the Apache back end. This means that request >> only passed onto Apache when data is available and so Apache can do >> its job quickly and not be tied up with dealing with slow HTTP >> request. Thus less risk of request being interrupted if process does >> indeed fall within that 5 seconds. Only passing on request when >> request data available, also means you will get better utilisation >> from Apache processes/threads and can configure it for less, thus >> reducing its memory overhead. >> >> Right now I can't find the part of the nginx documentation that talks >> about request buffering. The proxy documentation tends only to talk >> about response buffering as far as configuration parameters. >> >> > I though that using maximum-requests could prevent possible memory >> > leaks and exessive memory consumption. >> > Isn't it a right supposition? >> >> It can, but as described can cause conflict with long running >> uploads/requests if they are greater than default shutdown timeout of >> 5 seconds. You can adjust the shutdown timeout using shutdown-timeout >> option to WSGIDaemonProcess, but make it too long and you risk >> perceived delays by user if all daemon mode processes in group restart >> about the same time. >> >> > And also if I understand it right I will get the same errors on >> > application update when my WSGI application will be restarted. >> >> Yes, the shutdown timeout comes into play on any self restart of >> mod_wsgi daemon processes. >> >> The only time that shutdown timeout doesn't apply is when you do a >> full Apache 'restart' or 'graceful'. In that case Apache itself >> applies a 3 second timeout and will forcibly kill the mod_wsgi daemon >> mode processes after that. Can't override that specific Apache >> timeout. >> >> > Do you have any thoughts how could I find any misbehaving long-running >> > process with a stacktrace? >> >> One could use WSGI wrappers around your application object to add a >> request timer, but would first contemplate on whether it is the upload >> time for file rather than processing time. >> >> Graham >> >> > Best regards, >> > Kirill Zaborsky >> >> > On Jan 8, 1:06 am, Graham Dumpleton <[email protected]> >> > wrote: >> >> 2010/1/8 qrilka <[email protected]>: >> >> >> > From VirtualHost specific log: >> >> > ------------------------ >> >> > [Thu Jan 07 21:09:49 2010] [info] mod_wsgi (pid=12366): Maximum >> >> > requests reached 'av_factory'. >> >> > [Thu Jan 07 21:09:49 2010] [info] mod_wsgi (pid=12366): Shutdown >> >> > requested 'av_factory'. >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=12366): Aborting >> >> > process 'av_factory'. >> >> >> This line indicates that what I described previously is occurring and >> >> is likely the cause. >> >> >> That is, when reaching maximum-requests, there are long running >> >> requests or stuck requests that don't complete within the default 5 >> >> second window for shutting down a daemon process. >> >> >> When that occurs, even though still running the process is forcibly >> >> exited, even without shutting down Python interpreter properly. As a >> >> result, the Apache server child process which is proxying that >> >> specific request to the mod_wsgi daemon mode process sees the >> >> connection to daemon process abruptly cut off and as such you may see >> >> errors about premature end of script headers or the various filter >> >> errors or broken pipe messages depending on where a request was up to. >> >> >> Do you have any idea about whether you legitimately have requests that >> >> take longer than 5 seconds to process? >> >> >> For what reason are you using maximum-requests in the first place? If >> >> you don't have to use that option for some reason, the issue should be >> >> avoided. >> >> >> Graham> [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=28423): Attach >> >> > interpreter ''. >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=28423): Enable monitor >> >> > thread in process 'av_factory'. >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=28423): Enable >> >> > deadlock thread in process 'av_factory'. >> >> > [Thu Jan 07 21:09:54 2010] [info] [client 188.113.58.162] mod_wsgi >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "modwsgi" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/modwsgi?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > > >
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
