2010/1/13 qrilka <[email protected]>: > No, I don't see anything like that (at least at the time of the latest > 'Premature end' message). > Nginx upload module [1] requires some extra work and testing so it is > not on the production server yet.
FWIW, the benefits of using nginx in front to isolate Apache from slow clients doesn't require that module. What I was talking about is simply an aspect of how the standard nginx mod_proxy implementation works. Graham > I will let know if there will be any new details on this issue. > BTW it seems to be quite unlikely that I have some 'special' web > application which requires something like a voodoo magic and I don't > find any reports about such problems in the internet. It appears to me > to be some kind of server misconfiguration but I don't see any new > places where that misconfiguration could be. > > [1] http://www.grid.net.ru/nginx/upload.en.html > > Thanks once again. > > Best regards, > Kirill Zaborsky > > On Jan 13, 2:49 am, Graham Dumpleton <[email protected]> > wrote: >> 2010/1/13 qrilka <[email protected]>: >> >> > There is no CGI scripts on that server. >> > Just some PHP sites and Django. >> > And the message appears in Django virtual host log. >> >> Be aware that if this is being caused by a process crash, then the >> Segmentation Fault message will appear in the main Apache error log >> and not in the virtual host specific error log as is Apache parent >> monitoring process which is monitoring the fact the process died and >> it isn't linked to a specific virtual host. >> >> So, ensure you are paying attention to any error messages in the main >> Apache error log at the same time, including those which aren't >> specifically tagged as coming from mod_wsgi. If you find anything of >> relevance, then let me know. >> >> Graham >> >> > Best regards, >> > Kirill Zaborsky >> >> > On Jan 13, 1:33 am, Graham Dumpleton <[email protected]> >> > wrote: >> >> 2010/1/13 qrilka <[email protected]>: >> >> >> > Nobody else is doing anything with Apache but still I see the same >> >> > errors. >> >> > Today I'll try to setup nginx to see if it will help. >> >> >> The message 'Premature end of script headers' also gets generated by >> >> broken CGI scripts. So, you need to look quite closely at error logs >> >> and ensure that they are in fact linked to a WSGI request. >> >> >> Graham >> >> >> > Best regards, >> >> > Kirill Zaborsky >> >> >> > On Jan 12, 1:39 am, Graham Dumpleton <[email protected]> >> >> > wrote: >> >> >> 2010/1/11 qrilka <[email protected]>: >> >> >> >> > Thanks for your reply, Graham. >> >> >> > It was quite clear before that I need to spend more time on proper >> >> >> > server configuration (e.g. Apache now isn't quite tuned at least I >> >> >> > need to separate python from PHP). I will look into you proposal on >> >> >> > using nginx for proxying long-running requests. >> >> >> > BTW I removed maximum-requests but still see 'Premature end of script >> >> >> > headers' messages in error log. >> >> >> > What could be a reason for that? >> >> >> > Are there any other restarts for mod_wsgi except changed wsgi file >> >> >> > and >> >> >> > maximum-requests option? >> >> >> >> Could also occur if someone is doing an: >> >> >> >> apachectl graceful >> >> >> >> Ie., Apache graceful restart. >> >> >> >> Can occur here because Apache will allow its own server child process >> >> >> to keep running until active requests complete, but due to way APR >> >> >> library handles the other processes such as mod_wsgi daemon processes, >> >> >> it will kill them off regardless after 3 seconds. Thus, mod_wsgi >> >> >> daemon process goes away and the Apache server child process proxying >> >> >> request to the daemon process will then see connection close and get >> >> >> that error message. >> >> >> >> Presumably an Apache graceful shutdown could cause something similar. >> >> >> >> The question is therefore if anyone is doing graceful restarts on >> >> >> Apache at same time as you still see the message. >> >> >> >> The only other times that error message could arise is if the mod_wsgi >> >> >> daemon process crashed, or if someone has sent an explicit signal to >> >> >> the mod_wsgi daemon process to make it shutdown. >> >> >> >> Graham >> >> >> >> > Many thanks for you help. >> >> >> >> > Best regards, >> >> >> > Kirill Zaborsky >> >> >> >> > On Jan 11, 3:23 am, Graham Dumpleton <[email protected]> >> >> >> > wrote: >> >> >> >> 2010/1/8 qrilka <[email protected]>: >> >> >> >> >> > OK, I'll investigate how that could be solved. >> >> >> >> > Actually application is very simple and the only problem could be >> >> >> >> > with >> >> >> >> > some operations on large images. >> >> >> >> > But I do not see how that could lead to some operations taking >> >> >> >> > more >> >> >> >> > than 1 second. >> >> >> >> >> That time of 5 seconds isn't just for processing of the image but >> >> >> >> also >> >> >> >> inclusive of the time it takes to upload the image to the >> >> >> >> application >> >> >> >> if what you are doing first involves an upload. >> >> >> >> >> Thus, if dealing with large images or slow HTTP clients and clients >> >> >> >> are talking direct to Apache, then you may well exceed that time. >> >> >> >> >> What you can do to partly isolate yourself from problem of slow HTTP >> >> >> >> clients is to put nginx proxy in front of Apache. At least for files >> >> >> >> up to some default, nginx will buffer the upload before actually >> >> >> >> triggering the proxy to the Apache back end. This means that request >> >> >> >> only passed onto Apache when data is available and so Apache can do >> >> >> >> its job quickly and not be tied up with dealing with slow HTTP >> >> >> >> request. Thus less risk of request being interrupted if process does >> >> >> >> indeed fall within that 5 seconds. Only passing on request when >> >> >> >> request data available, also means you will get better utilisation >> >> >> >> from Apache processes/threads and can configure it for less, thus >> >> >> >> reducing its memory overhead. >> >> >> >> >> Right now I can't find the part of the nginx documentation that >> >> >> >> talks >> >> >> >> about request buffering. The proxy documentation tends only to talk >> >> >> >> about response buffering as far as configuration parameters. >> >> >> >> >> > I though that using maximum-requests could prevent possible memory >> >> >> >> > leaks and exessive memory consumption. >> >> >> >> > Isn't it a right supposition? >> >> >> >> >> It can, but as described can cause conflict with long running >> >> >> >> uploads/requests if they are greater than default shutdown timeout >> >> >> >> of >> >> >> >> 5 seconds. You can adjust the shutdown timeout using >> >> >> >> shutdown-timeout >> >> >> >> option to WSGIDaemonProcess, but make it too long and you risk >> >> >> >> perceived delays by user if all daemon mode processes in group >> >> >> >> restart >> >> >> >> about the same time. >> >> >> >> >> > And also if I understand it right I will get the same errors on >> >> >> >> > application update when my WSGI application will be restarted. >> >> >> >> >> Yes, the shutdown timeout comes into play on any self restart of >> >> >> >> mod_wsgi daemon processes. >> >> >> >> >> The only time that shutdown timeout doesn't apply is when you do a >> >> >> >> full Apache 'restart' or 'graceful'. In that case Apache itself >> >> >> >> applies a 3 second timeout and will forcibly kill the mod_wsgi >> >> >> >> daemon >> >> >> >> mode processes after that. Can't override that specific Apache >> >> >> >> timeout. >> >> >> >> >> > Do you have any thoughts how could I find any misbehaving >> >> >> >> > long-running >> >> >> >> > process with a stacktrace? >> >> >> >> >> One could use WSGI wrappers around your application object to add a >> >> >> >> request timer, but would first contemplate on whether it is the >> >> >> >> upload >> >> >> >> time for file rather than processing time. >> >> >> >> >> Graham >> >> >> >> >> > Best regards, >> >> >> >> > Kirill Zaborsky >> >> >> >> >> > On Jan 8, 1:06 am, Graham Dumpleton <[email protected]> >> >> >> >> > wrote: >> >> >> >> >> 2010/1/8 qrilka <[email protected]>: >> >> >> >> >> >> > From VirtualHost specific log: >> >> >> >> >> > ------------------------ >> >> >> >> >> > [Thu Jan 07 21:09:49 2010] [info] mod_wsgi (pid=12366): Maximum >> >> >> >> >> > requests reached 'av_factory'. >> >> >> >> >> > [Thu Jan 07 21:09:49 2010] [info] mod_wsgi (pid=12366): >> >> >> >> >> > Shutdown >> >> >> >> >> > requested 'av_factory'. >> >> >> >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=12366): >> >> >> >> >> > Aborting >> >> >> >> >> > process 'av_factory'. >> >> >> >> >> >> This line indicates that what I described previously is >> >> >> >> >> occurring and >> >> >> >> >> is likely the cause. >> >> >> >> >> >> That is, when reaching maximum-requests, there are long running >> >> >> >> >> requests or stuck requests that don't complete within the >> >> >> >> >> default 5 >> >> >> >> >> second window for shutting down a daemon process. >> >> >> >> >> >> When that occurs, even though still running the process is >> >> >> >> >> forcibly >> >> >> >> >> exited, even without shutting down Python interpreter properly. >> >> >> >> >> As a >> >> >> >> >> result, the Apache server child process which is proxying that >> >> >> >> >> specific request to the mod_wsgi daemon mode process sees the >> >> >> >> >> connection to daemon process abruptly cut off and as such you >> >> >> >> >> may see >> >> >> >> >> errors about premature end of script headers or the various >> >> >> >> >> filter >> >> >> >> >> errors or broken pipe messages depending on where a request was >> >> >> >> >> up to. >> >> >> >> >> >> Do you have any idea about whether you legitimately have >> >> >> >> >> requests that >> >> >> >> >> take longer than 5 seconds to process? >> >> >> >> >> >> For what reason are you using maximum-requests in the first >> >> >> >> >> place? If >> >> >> >> >> you don't have to use that option for some reason, the issue >> >> >> >> >> should be >> >> >> >> >> avoided. >> >> >> >> >> >> Graham> [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=28423): >> >> >> >> >> Attach >> >> >> >> >> > interpreter ''. >> >> >> >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=28423): Enable >> >> >> >> >> > monitor >> >> >> >> >> > thread in process 'av_factory'. >> >> >> >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=28423): Enable >> >> >> >> >> > deadlock thread in process 'av_factory'. >> >> >> >> >> > [Thu Jan 07 21:09:54 2010] [info] [client 188.113.58.162] >> >> >> >> >> > mod_wsgi >> >> >> >> >> > -- >> >> >> >> > You received this message because you are subscribed to the >> >> >> >> > Google Groups "modwsgi" group. >> >> >> >> > To post to this group, send email to [email protected]. >> >> >> >> > To unsubscribe from this group, send email to >> >> >> >> > [email protected]. >> >> >> >> > For more options, visit this group >> >> >> >> > athttp://groups.google.com/group/modwsgi?hl=en. >> >> >> >> > -- >> >> >> > You received this message because you are subscribed to the Google >> >> >> > Groups "modwsgi" group. >> >> >> > To post to this group, send email to [email protected]. >> >> >> > To unsubscribe from this group, send email to >> >> >> > [email protected]. >> >> >> > For more options, visit this group >> >> >> > athttp://groups.google.com/group/modwsgi?hl=en. >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "modwsgi" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group >> >> > athttp://groups.google.com/group/modwsgi?hl=en. >> >> > -- >> > You received this message because you are subscribed to the >> >> ... >> >> read more » > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > > >
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
