2010/1/14 qrilka <[email protected]>: > I could not find anything about uploads proxying in nginx docs. > Did I miss something?
The mod_proxy module doesn't distinguish types of requests, any HTTP request can have a body content and all mod_proxy does is pass that data through to the backend as is for the original web application to process. The module you are talking about actually tries to interpret the data and splits it into files as I understand it. The web application would then need to be modified somewhat to be able to pick up special headers about where uploaded files were placed. As such, that module will be incompatible with an existing Python web frameworks file upload mechanisms. Graham > Anyways looks like we will try that module as it seem to work OK (but > some hours later) > > Best regards, > Kirill Zaborsky > > On Jan 13, 1:00 pm, Graham Dumpleton <[email protected]> > wrote: >> 2010/1/13 qrilka <[email protected]>: >> >> > No, I don't see anything like that (at least at the time of the latest >> > 'Premature end' message). >> > Nginx upload module [1] requires some extra work and testing so it is >> > not on the production server yet. >> >> FWIW, the benefits of using nginx in front to isolate Apache from slow >> clients doesn't require that module. What I was talking about is >> simply an aspect of how the standard nginx mod_proxy implementation >> works. >> >> Graham >> >> > I will let know if there will be any new details on this issue. >> > BTW it seems to be quite unlikely that I have some 'special' web >> > application which requires something like a voodoo magic and I don't >> > find any reports about such problems in the internet. It appears to me >> > to be some kind of server misconfiguration but I don't see any new >> > places where that misconfiguration could be. >> >> > [1]http://www.grid.net.ru/nginx/upload.en.html >> >> > Thanks once again. >> >> > Best regards, >> > Kirill Zaborsky >> >> > On Jan 13, 2:49 am, Graham Dumpleton <[email protected]> >> > wrote: >> >> 2010/1/13 qrilka <[email protected]>: >> >> >> > There is no CGI scripts on that server. >> >> > Just some PHP sites and Django. >> >> > And the message appears in Django virtual host log. >> >> >> Be aware that if this is being caused by a process crash, then the >> >> Segmentation Fault message will appear in the main Apache error log >> >> and not in the virtual host specific error log as is Apache parent >> >> monitoring process which is monitoring the fact the process died and >> >> it isn't linked to a specific virtual host. >> >> >> So, ensure you are paying attention to any error messages in the main >> >> Apache error log at the same time, including those which aren't >> >> specifically tagged as coming from mod_wsgi. If you find anything of >> >> relevance, then let me know. >> >> >> Graham >> >> >> > Best regards, >> >> > Kirill Zaborsky >> >> >> > On Jan 13, 1:33 am, Graham Dumpleton <[email protected]> >> >> > wrote: >> >> >> 2010/1/13 qrilka <[email protected]>: >> >> >> >> > Nobody else is doing anything with Apache but still I see the same >> >> >> > errors. >> >> >> > Today I'll try to setup nginx to see if it will help. >> >> >> >> The message 'Premature end of script headers' also gets generated by >> >> >> broken CGI scripts. So, you need to look quite closely at error logs >> >> >> and ensure that they are in fact linked to a WSGI request. >> >> >> >> Graham >> >> >> >> > Best regards, >> >> >> > Kirill Zaborsky >> >> >> >> > On Jan 12, 1:39 am, Graham Dumpleton <[email protected]> >> >> >> > wrote: >> >> >> >> 2010/1/11 qrilka <[email protected]>: >> >> >> >> >> > Thanks for your reply, Graham. >> >> >> >> > It was quite clear before that I need to spend more time on proper >> >> >> >> > server configuration (e.g. Apache now isn't quite tuned at least I >> >> >> >> > need to separate python from PHP). I will look into you proposal >> >> >> >> > on >> >> >> >> > using nginx for proxying long-running requests. >> >> >> >> > BTW I removed maximum-requests but still see 'Premature end of >> >> >> >> > script >> >> >> >> > headers' messages in error log. >> >> >> >> > What could be a reason for that? >> >> >> >> > Are there any other restarts for mod_wsgi except changed wsgi >> >> >> >> > file and >> >> >> >> > maximum-requests option? >> >> >> >> >> Could also occur if someone is doing an: >> >> >> >> >> apachectl graceful >> >> >> >> >> Ie., Apache graceful restart. >> >> >> >> >> Can occur here because Apache will allow its own server child >> >> >> >> process >> >> >> >> to keep running until active requests complete, but due to way APR >> >> >> >> library handles the other processes such as mod_wsgi daemon >> >> >> >> processes, >> >> >> >> it will kill them off regardless after 3 seconds. Thus, mod_wsgi >> >> >> >> daemon process goes away and the Apache server child process >> >> >> >> proxying >> >> >> >> request to the daemon process will then see connection close and get >> >> >> >> that error message. >> >> >> >> >> Presumably an Apache graceful shutdown could cause something >> >> >> >> similar. >> >> >> >> >> The question is therefore if anyone is doing graceful restarts on >> >> >> >> Apache at same time as you still see the message. >> >> >> >> >> The only other times that error message could arise is if the >> >> >> >> mod_wsgi >> >> >> >> daemon process crashed, or if someone has sent an explicit signal to >> >> >> >> the mod_wsgi daemon process to make it shutdown. >> >> >> >> >> Graham >> >> >> >> >> > Many thanks for you help. >> >> >> >> >> > Best regards, >> >> >> >> > Kirill Zaborsky >> >> >> >> >> > On Jan 11, 3:23 am, Graham Dumpleton <[email protected]> >> >> >> >> > wrote: >> >> >> >> >> 2010/1/8 qrilka <[email protected]>: >> >> >> >> >> >> > OK, I'll investigate how that could be solved. >> >> >> >> >> > Actually application is very simple and the only problem could >> >> >> >> >> > be with >> >> >> >> >> > some operations on large images. >> >> >> >> >> > But I do not see how that could lead to some operations taking >> >> >> >> >> > more >> >> >> >> >> > than 1 second. >> >> >> >> >> >> That time of 5 seconds isn't just for processing of the image >> >> >> >> >> but also >> >> >> >> >> inclusive of the time it takes to upload the image to the >> >> >> >> >> application >> >> >> >> >> if what you are doing first involves an upload. >> >> >> >> >> >> Thus, if dealing with large images or slow HTTP clients and >> >> >> >> >> clients >> >> >> >> >> are talking direct to Apache, then you may well exceed that time. >> >> >> >> >> >> What you can do to partly isolate yourself from problem of slow >> >> >> >> >> HTTP >> >> >> >> >> clients is to put nginx proxy in front of Apache. At least for >> >> >> >> >> files >> >> >> >> >> up to some default, nginx will buffer the upload before actually >> >> >> >> >> triggering the proxy to the Apache back end. This means that >> >> >> >> >> request >> >> >> >> >> only passed onto Apache when data is available and so Apache can >> >> >> >> >> do >> >> >> >> >> its job quickly and not be tied up with dealing with slow HTTP >> >> >> >> >> request. Thus less risk of request being interrupted if process >> >> >> >> >> does >> >> >> >> >> indeed fall within that 5 seconds. Only passing on request when >> >> >> >> >> request data available, also means you will get better >> >> >> >> >> utilisation >> >> >> >> >> from Apache processes/threads and can configure it for less, thus >> >> >> >> >> reducing its memory overhead. >> >> >> >> >> >> Right now I can't find the part of the nginx documentation that >> >> >> >> >> talks >> >> >> >> >> about request buffering. The proxy documentation tends only to >> >> >> >> >> talk >> >> >> >> >> about response buffering as far as configuration parameters. >> >> >> >> >> >> > I though that using maximum-requests could prevent possible >> >> >> >> >> > memory >> >> >> >> >> > leaks and exessive memory consumption. >> >> >> >> >> > Isn't it a right supposition? >> >> >> >> >> >> It can, but as described can cause conflict with long running >> >> >> >> >> uploads/requests if they are greater than default shutdown >> >> >> >> >> timeout of >> >> >> >> >> 5 seconds. You can adjust the shutdown timeout using >> >> >> >> >> shutdown-timeout >> >> >> >> >> option to WSGIDaemonProcess, but make it too long and you risk >> >> >> >> >> perceived delays by user if all daemon mode processes in group >> >> >> >> >> restart >> >> >> >> >> about the same time. >> >> >> >> >> >> > And also if I understand it right I will get the same errors on >> >> >> >> >> > application update when my WSGI application will be restarted. >> >> >> >> >> >> Yes, the shutdown timeout comes into play on any self restart of >> >> >> >> >> mod_wsgi daemon processes. >> >> >> >> >> >> The only time that shutdown timeout doesn't apply is when you do >> >> >> >> >> a >> >> >> >> >> full Apache 'restart' or 'graceful'. In that case Apache itself >> >> >> >> >> applies a 3 second timeout and will forcibly kill the mod_wsgi >> >> >> >> >> daemon >> >> >> >> >> mode processes after that. Can't override that specific Apache >> >> >> >> >> timeout. >> >> >> >> >> >> > Do you have any thoughts how could I find any misbehaving >> >> >> >> >> > long-running >> >> >> >> >> > process with a stacktrace? >> >> >> >> >> >> One could use WSGI wrappers around your application object to >> >> >> >> >> add a >> >> >> >> >> request timer, but would first contemplate on whether it is the >> >> >> >> >> upload >> >> >> >> >> time for file rather than processing time. >> >> >> >> >> >> Graham >> >> >> >> >> >> > Best regards, >> >> >> >> >> > Kirill Zaborsky >> >> >> >> >> >> > On Jan 8, 1:06 am, Graham Dumpleton >> >> >> >> >> > <[email protected]> >> >> >> >> >> > wrote: >> >> >> >> >> >> 2010/1/8 qrilka <[email protected]>: >> >> >> >> >> >> >> > From VirtualHost specific log: >> >> >> >> >> >> > ------------------------ >> >> >> >> >> >> > [Thu Jan 07 21:09:49 2010] [info] mod_wsgi (pid=12366): >> >> >> >> >> >> > Maximum >> >> >> >> >> >> > requests reached 'av_factory'. >> >> >> >> >> >> > [Thu Jan 07 21:09:49 2010] [info] mod_wsgi (pid=12366): >> >> >> >> >> >> > Shutdown >> >> >> >> >> >> > requested 'av_factory'. >> >> >> >> >> >> > [Thu Jan 07 21:09:54 2010] [info] mod_wsgi (pid=12366): >> >> >> >> >> >> > Aborting >> >> >> >> >> >> > process 'av_factory'. >> >> >> >> >> >> >> This line indicates that what I described previously is >> >> >> >> >> >> occurring and >> >> >> >> >> >> is likely the cause. >> >> >> >> >> >> >> That is, when reaching maximum-requests, there are long >> >> >> >> >> >> running >> >> >> >> >> >> requests or stuck requests that don't complete within the >> >> >> >> >> >> default 5 >> >> >> >> >> >> second window for shutting down a daemon process. >> >> >> >> >> >> >> When that occurs, even though still running the process is >> >> >> >> >> >> forcibly >> >> >> >> >> >> exited, even without shutting down Python interpreter >> >> >> >> >> >> properly. As a >> >> >> >> >> >> result, the Apache server child process which is proxying that >> >> >> >> >> >> specific request to the mod_wsgi daemon mode process sees the >> >> >> >> >> >> connection to daemon process abruptly cut off and as such you >> >> >> >> >> >> may see >> >> >> >> >> >> errors about premature end of script headers or the various >> >> >> >> >> >> filter >> >> >> >> >> >> errors or broken pipe messages depending on where a request >> >> >> >> >> >> was up to. >> >> >> >> >> >> >> Do you have any idea about whether you legitimately have >> >> >> >> >> >> requests that >> >> >> >> >> >> take longer than 5 seconds to process? >> >> >> >> >> >> >> For what reason are you using maximum-requests in the first >> >> >> >> >> >> place? If >> >> >> >> >> >> you don't have to use that option for some reason, the issue >> >> >> >> >> >> should be >> >> >> >> >> >> avoided. >> >> >> >> >> >> >> Graham> [Thu Jan 07 21:09:54 2010] [info] mod_wsgi >> >> >> >> >> >> (pid=28423): Attach >> >> >> >> >> >> > interpreter ''. >> >> >> >> >> >> > [Thu Jan 07 >> >> ... >> >> read more » > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > > >
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
