Should now be fixed. On 03/02/2015, at 8:50 PM, Graham Dumpleton <[email protected]> wrote:
> The application of the eviction timeout should not be fixed in develop branch. > > https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.tar.gz > > Graham > > On 03/02/2015, at 5:02 PM, Graham Dumpleton <[email protected]> > wrote: > >> >> >> On 3 February 2015 at 04:15, Kent Bower <[email protected]> wrote: >> On Sun, Feb 1, 2015 at 7:08 PM, Graham Dumpleton >> <[email protected]> wrote: >> Your Flask client doesn't need to know about Celery, as your web application >> accepts requests as normal and it is your Python code which would queue the >> job with Celery. >> >> Now looking back, the only configuration I can find, but which I don't know >> if it is your actual production configuration is: >> >> WSGIDaemonProcess rarch processes=3 threads=2 inactivity-timeout=1800 >> display-name=%{GROUP} graceful-timeout=140 eviction-timeout=60 >> python-eggs=/home/rarch/tg2env/lib/python-egg-cache >> >> Provided that you don't then start to have overall host memory issues, the >> simplest way around this whole issue is not to use a multithreaded process. >> >> What you would do is vertically partition your URL name space so that just >> the URLs which do the long running report generation would be delegated to >> single threaded processes. Everything else would keep going to the >> multithread processes. >> >> WSGIDaemonProcess rarch processes=3 threads=2 >> WSGIDaemonProvess rarch-long-running processes=6 threads=1 >> maximum-requests=20 >> >> WSGIProcessGroup rarch >> >> <Location /suburl/of/long/running/report/generator> >> WSGIProcessGroup rarch-long-running >> </Location> >> >> You wouldn't even have to worry about the graceful-timeout on >> rarch-long-running as that is only relevant for maxiumum-requests where it >> is a multithreaded processes. >> >> So what would happen is that when the request has finished, if >> maximum-requests is reached, the process would be restarted even before any >> new request was accepted by the process, so there is no chance of a new >> request being interrupted. >> >> You could still set an eviction-timeout of some suitably large value to >> allow you to use SIGUSR1 to be sent to processes in that daemon process >> group to shut them down. >> >> In this case, having eviction-timeout being able to be set independent of >> graceful-timeout (for maximum-requests), is probably useful and so I will >> retain the option. >> >> So is there any reason you couldn't use a daemon process group with many >> single threaded process instead? >> >> >> This is very good to know (that single threaded procs would behave more >> ideally in these circumstances). The above was just my configuration for >> testing 'eviction-timeout'. Our software generally runs with many more >> processes and threads, on servers with maybe 16 or 32 GB RAM. And >> unfortunately, the RAM is the limiting resource here as our python app, >> built on turbo-gears, is a memory hog and we have yet to find the resources >> to dissect that. I was aiming to head in the direction of URL partitioning, >> but there are big obstacles. (Chiefly, RAM consumption would make threads=1 >> and yet more processes very difficult unless we spend the huge effort in >> dissecting the app to locate and pull the many unused memory hogging >> libraries out.) >> >> So, URL partitioning is sort of the ideal, distant solution, as well as a >> Celery-like polling solution, but out of my reach for now. >> >> Have you ever run a test where you compare the whole memory usage of your >> application where all URLs are visited, to how much memory is used if only >> the URL which generates the long running report is visited? >> >> In Django at least, a lot of stuff is lazily loaded only when a URL >> requiring it is first accessed. So even with a heavy code base, there can >> still be benefits in splitting out URLs to their own processes because the >> whole code base wouldn't be loaded due to the lazy loading. >> >> So do you have any actual memory figures from doing that? >> >> How many URLs are there that generates these reports vs those that don't, or >> is that all the whole application does? >> >> Are your most frequently visited URLs those generating the reports or >> something else? >> >> Another question for multithreaded graceful-timeout with maximum-requests: >> during a period of heavy traffic, it seems the graceful-timeout setting just >> pushes the real timeout until shutdown-timeout because, if heavy enough, >> you'll be getting requests during graceful-timeout. That diminishes the >> fidelity of "graceful-timeout." Do you see where I'm coming from (even if >> you're happy with the design and don't want to mess with it, which I'd >> understand)? >> >> >> Ok, here is the log demonstrating the troubles I saw with eviction-timeout. >> For demonstration purposes, here is the simplified directive I'm using: >> >> WSGIDaemonProcess rarch processes=1 threads=1 display-name=%{GROUP} >> graceful-timeout=140 eviction-timeout=60 >> python-eggs=/home/rarch/tg2env/lib/python-egg-cache >> >> Here is the log: >> >> [Mon Feb 02 11:36:16 2015] [info] Init: Initializing (virtual) servers for >> SSL >> [Mon Feb 02 11:36:16 2015] [info] Server: Apache/2.2.3, Interface: >> mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5 >> [Mon Feb 02 11:36:16 2015] [notice] Digest: generating secret for digest >> authentication ... >> [Mon Feb 02 11:36:16 2015] [notice] Digest: done >> [Mon Feb 02 11:36:16 2015] [info] APR LDAP: Built with OpenLDAP LDAP SDK >> [Mon Feb 02 11:36:16 2015] [info] LDAP: SSL support available >> [Mon Feb 02 11:36:16 2015] [info] Init: Seeding PRNG with 256 bytes of >> entropy >> [Mon Feb 02 11:36:16 2015] [info] Init: Generating temporary RSA private >> keys (512/1024 bits) >> [Mon Feb 02 11:36:16 2015] [info] Init: Generating temporary DH parameters >> (512/1024 bits) >> [Mon Feb 02 11:36:16 2015] [info] Shared memory session cache initialised >> [Mon Feb 02 11:36:16 2015] [info] Init: Initializing (virtual) servers for >> SSL >> [Mon Feb 02 11:36:16 2015] [info] Server: Apache/2.2.3, Interface: >> mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5 >> [Mon Feb 02 11:36:16 2015] [info] mod_wsgi (pid=29447): Starting process >> 'rarch' with uid=48, gid=48 and threads=1. >> [Mon Feb 02 11:36:16 2015] [info] mod_wsgi (pid=29447): Python home >> /home/rarch/tg2env. >> [Mon Feb 02 11:36:16 2015] [info] mod_wsgi (pid=29447): Initializing Python. >> [Mon Feb 02 11:36:16 2015] [notice] Apache/2.2.3 (CentOS) configured -- >> resuming normal operations >> [Mon Feb 02 11:36:16 2015] [info] Server built: Aug 30 2010 12:28:40 >> [Mon Feb 02 11:36:16 2015] [info] mod_wsgi (pid=29447): Attach interpreter >> ''. >> [Mon Feb 02 11:36:16 2015] [info] mod_wsgi (pid=29447, process='rarch', >> application=''): Loading WSGI script >> '/home/rarch/trunk/src/appserver/wsgi-config/wsgi-deployment.py'. >> [Mon Feb 02 11:39:13 2015] [info] mod_wsgi (pid=29447): Process eviction >> requested, waiting for requests to complete 'rarch'. >> [Mon Feb 02 11:41:00 2015] [info] mod_wsgi (pid=29447): Daemon process >> graceful timer expired 'rarch'. >> [Mon Feb 02 11:41:00 2015] [info] mod_wsgi (pid=29447): Shutdown requested >> 'rarch'. >> [Mon Feb 02 11:41:05 2015] [info] mod_wsgi (pid=29447): Aborting process >> 'rarch'. >> [Mon Feb 02 11:41:05 2015] [info] mod_wsgi (pid=29447): Exiting process >> 'rarch'. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=29447): Process 'rarch' has >> died, deregister and restart it. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=29447): Process 'rarch' has >> been deregistered and will no longer be monitored. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=31331): Starting process >> 'rarch' with uid=48, gid=48 and threads=1. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=31331): Python home >> /home/rarch/tg2env. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=31331): Initializing Python. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=31331): Attach interpreter >> ''. >> [Mon Feb 02 11:41:06 2015] [info] mod_wsgi (pid=31331, process='rarch', >> application=''): Loading WSGI script >> '/home/rarch/trunk/src/appserver/wsgi-config/wsgi-deployment.py'. >> >> The process was signaled at 11:39:13 with eviction-timeout=60 but 11:40:13 >> came and passed and nothing happened until 107 seconds passed, at which time >> graceful timer expired. >> >> >> Next, I changed the parameters a little: >> >> WSGIDaemonProcess rarch processes=1 threads=1 display-name=%{GROUP} >> eviction-timeout=30 graceful-timeout=240 >> python-eggs=/home/rarch/tg2env/lib/python-egg-cache >> >> [Mon Feb 02 12:06:57 2015] [info] mod_wsgi (pid=3381): Starting process >> 'rarch' with uid=48, gid=48 and threads=1. >> [Mon Feb 02 12:06:57 2015] [info] mod_wsgi (pid=3381): Python home >> /home/rarch/tg2env. >> [Mon Feb 02 12:06:57 2015] [info] mod_wsgi (pid=3381): Initializing Python. >> [Mon Feb 02 12:06:57 2015] [notice] Apache/2.2.3 (CentOS) configured -- >> resuming normal operations >> [Mon Feb 02 12:06:57 2015] [info] Server built: Aug 30 2010 12:28:40 >> [Mon Feb 02 12:06:57 2015] [info] mod_wsgi (pid=3381): Attach interpreter ''. >> [Mon Feb 02 12:06:57 2015] [info] mod_wsgi (pid=3381, process='rarch', >> application=''): Loading WSGI script >> '/home/rarch/trunk/src/appserver/wsgi-config/wsgi-deployment.py'. >> [Mon Feb 02 12:07:19 2015] [info] mod_wsgi (pid=3381): Process eviction >> requested, waiting for requests to complete 'rarch'. >> [Mon Feb 02 12:11:01 2015] [info] mod_wsgi (pid=3381): Daemon process >> graceful timer expired 'rarch'. >> [Mon Feb 02 12:11:01 2015] [info] mod_wsgi (pid=3381): Shutdown requested >> 'rarch'. >> [Mon Feb 02 12:11:06 2015] [info] mod_wsgi (pid=3381): Aborting process >> 'rarch'. >> [Mon Feb 02 12:11:06 2015] [info] mod_wsgi (pid=3381): Exiting process >> 'rarch'. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=3381): Process 'rarch' has >> died, deregister and restart it. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=3381): Process 'rarch' has >> been deregistered and will no longer be monitored. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=7028): Starting process >> 'rarch' with uid=48, gid=48 and threads=1. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=7028): Python home >> /home/rarch/tg2env. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=7028): Initializing Python. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=7028): Attach interpreter ''. >> [Mon Feb 02 12:11:07 2015] [info] mod_wsgi (pid=7028, process='rarch', >> application=''): Loading WSGI script >> '/home/rarch/trunk/src/appserver/wsgi-config/wsgi-deployment.py'. >> >> >> So, for me, eviction-timeout is apparently being ignored... >> >> The background monitor thread which monitors for expiry wasn't taking into >> consideration the eviction timeout period being able to be less than the >> graceful timeout. I didn't see a problem as I was also setting request >> timeout, which causes the way the monitor thread works to be different, >> waking up every second regardless. I will work on a fix for that. >> >> Another issue for consideration is if a graceful timeout is already in >> progress and a signal comes in for eviction, which timeout wins? Right now >> the eviction time will trump the graceful time if already set by maximum >> requests. The converse isn't true though in that if already in eviction >> cycle and maximum requests arrives, it wouldn't be trumped by graceful >> timeout. So eviction time had authority given that it was triggered by >> explicit user signal. It does mean that the signal could effectively extend >> what ever graceful time was in progress. >> >> Graham >> >> Thanks again for all your time and help, >> Kent >> >> >> >> Note that since only a sub set of URLs would go to the daemon process group, >> the memory usage profile will change as you aren't potentially loading the >> complete application code into those processes and only those needed for >> that URL and that report. So it could use up less memory than application as >> a whole, allowing you to have multiple single threaded processes with no >> issue. >> >> Graham >> >> On 31/01/2015, at 12:31 AM, Kent <[email protected]> wrote: >> >>> Thanks for your reply and recommendations. We're aware of the issues, but >>> I didn't give the full picture for brevity's sake. The reports are user >>> generated reports. Ultimately, the users know whether the reports should >>> return quickly (which many, many will), or whether they are long-running. >>> There is no way for the application to know that, so to avoid some sort of >>> polling (which we've done in the past and was a pain in the rear to users), >>> the design is to allow the user to decide whether to run the report in the >>> background or "foreground" via a check box. Since most reports will return >>> in a matter of a minute or so, we wanted to avoid the pain of making them >>> poll, but I need to look at Celery. However, I'm not comfortable punishing >>> users for accidentally choosing foreground on a long-running report. That >>> is, not for an automatic turn-over mechanism like maximum-requests or >>> inactivity-timeout. In my mind, those are inherently different than >>> something like a SIGUSR1 mechanism because the former are automatic. >>> >>> So, while admitting there are edge cases we are using that don't have a >>> perfect solution (or even admitting we need a better mechanism in that >>> case), it still seems to me mod_wsgi should be somewhat agnostic of design >>> choices. In other words, when it comes to automatic turning over of >>> processes, it seems mod_wsgi shouldn't be involved with length of time >>> considerations, except to allow the user to specify timeouts. See, the >>> long running reports are only one of my concerns: we also fight with >>> database locks sometimes, held by another application attached to the same >>> database and wholly out of our control. Sometimes those locks can be held >>> for many minutes on a request that normally should complete within seconds. >>> There too, it seems mod_wsgi should be very gentle in the automatic >>> turnover cases. >>> >>> Thanks for pointing to Celery. I really wonder whether I can get a message >>> broker to work with Adobe Flash, our current client, but I haven't looked >>> into this much yet. >>> >>> Also, my apologies if you believe this to have been a waste of time on your >>> part. You've been extremely helpful, though and I'm quite thankful for >>> your time! I understand you not wanting to redesign the shutdown-timeout >>> thing and mess with what otherwise isn't broken. Would you still like me >>> to post the apache debug logs regarding 'eviction-timeout' or have you >>> changed your mind about releasing that? (In which case, extra apologies.) >>> >>> Kent >>> >>> >>> >>> >>> On Friday, January 30, 2015 at 6:34:28 AM UTC-5, Graham Dumpleton wrote: >>> If you have web requests generating reports which take 40 minutes to run, >>> you are going the wrong way about it. >>> >>> What would be regarded as best practice for long running requests is to use >>> a task queuing system to queue up the task to be run and run it in a >>> distinct set of processes to the web server. Your web request can then >>> return immediately, with some sort of polling system used as necessary to >>> check the progress of the task and allow the result to be downloaded when >>> complete. By using a separate system to run the tasks, it doesn't matter >>> whether the web server is restarted as the tasks will still run and after >>> the web server is restarted, a user can still check on progress of the >>> tasks and get back his response. >>> >>> The most common such task execution system for doing this sort of thing is >>> Celery. >>> >>> So it is because you aren't using the correct tool for the job here that >>> you are fighting against things like timeouts in the web server. No web >>> server is really a suitable environment to be used as an in process task >>> execution system. The web server should handle requests quickly and offload >>> longer processing tasks a separate task system which is purpose built for >>> handling the management of long running tasks. >>> >>> I am not inclined to keep fiddling how the timeouts work now I understand >>> what you are trying to do. I am even questioning now whether I should have >>> introduced the separate eviction timeout I already did given that it is >>> turning out to be a questionable use case. >>> >>> I would really recommend you look at re-architecting how you do things. I >>> don't think I would have any trouble finding others on the list who would >>> advise the same thing and who could also give you further advice on using >>> something like Celery instead for task execution. >>> >>> Graham >>> >>> On 29/01/2015, at 7:30 AM, Kent <[email protected]> wrote: >>> >>> Ok, I plan to run those tests with debug and post, but please, in the >>> meantime: >>> >>> For our app, not interrupting existing requests is a higher priority than >>> being able to accept new requests, particularly since we typically run many >>> wsgi processes, each with a handful of threads. So, I'm not really >>> concerned about maintaining always available threads (statistically, I will >>> be fine... that isn't the issue for me). >>> >>> In these circumstances, it would be much better for all these triggering >>> events (SIGUSR1, maximum-requests, or inactivity-timeout, etc.) to >>> immediately stop accepting new requests and "concentrate" on shutting down. >>> (Unless that means requests waiting in apache are terminated because they >>> were queued for this particular process, but I doubt apache has already >>> determined the request's process if none are available, has it?) With high >>> graceful-timeout/eviction-timeout and low shutdown-timeout, I run a pretty >>> high risk of accepting a new request at the tail end of graceful-timeout or >>> eviction-timeout, only to have it basically doomed to ungraceful death >>> because many of our requests are long running (very often well over 5 or 10 >>> sec). >>> >>> I guess that's why, through experimentation with SIGUSR1 a few years back, >>> I ended up "graceful-timeout=5 shutdown-timeout=300" ... the opposite of >>> how it would default, because this works well when trying to signal these >>> to recycle themselves: they basically immediately stop accepting new >>> requests so your "guaranteed" graceful timeout is 300. It seems I have no >>> way to "guarantee" a very large graceful timeout for each and every >>> request, even if affected by maximum-requests or inactivity-timeout, and >>> specify a different (lower) one for SIGUSR1 because the only truly >>> guaranteed lifetime in seconds is "shutdown-timeout," is that accurate? >>> >>> The ideal for our app, which may accept certain request that run for >>> several minutes is this: >>> if maximum-requests or inactivity-timeout is hit, stop taking new requests >>> immediately and shutdown as soon as possible, but give existing requests >>> basically all the time they need to finish (say, up to 40 minutes (for >>> long-running db reports)). >>> if SIGUSR1, stop taking new requests immediately and shutdown as soon as >>> possible, but give existing requests a really good chance to complete, >>> maybe 3-5 minutes, but not the 40 minutes, because this is slightly more >>> urgent (was triggered manually and a user is monitoring/waiting for >>> turnover and wants new code in place) >>> I don't think I can accomplish the above if I understand the design >>> correctly because a request may have been accepted at the tail end of >>> graceful-timeout/eviction-timeout and so is only guaranteed a lifetime of >>> shutdown-timeout, regardless of what the trigger was (SIGUSR1 vs. >>> automatic). >>> >>> Is my understanding of this accurate? >>> >>> >>> >>> On Tuesday, January 27, 2015 at 9:48:01 PM UTC-5, Graham Dumpleton wrote: >>> Can you ensure that LogLevel is set to at least info and provide what >>> messages are in the Apache error log file >>> >>> If I use: >>> >>> $ mod_wsgi-express start-server hack/sleep.wsg--log-level=debug >>> --verbose-debugging --eviction-timeout 30 --graceful-timeout 60 >>> >>> which is equivalent to: >>> >>> WSGIDaemonProcess … graceful-timeout=60 eviction-timeout=30 >>> >>> and fire a request against application that sleeps a long time I see in the >>> Apache error logs at the time of the signal: >>> >>> [Wed Jan 28 13:34:34 2015] [info] mod_wsgi (pid=29639): Process eviction >>> requested, waiting for requests to complete 'localhost:8000'. >>> >>> At the end of the 30 seconds given by the eviction timeout I see: >>> >>> [Wed Jan 28 13:35:05 2015] [info] mod_wsgi (pid=29639): Daemon process >>> graceful timer expired 'localhost:8000'. >>> [Wed Jan 28 13:35:05 2015] [info] mod_wsgi (pid=29639): Shutdown requested >>> 'localhost:8000'. >>> >>> Up till that point the process would still have been accepting new requests >>> and was waiting for point that there was no active requests to allow it to >>> shutdown. >>> >>> As the timeout tripped at 30 seconds, it then instead goes into the more >>> brutal shutdown process. No new requests are accepted from this point. >>> >>> For my setup the shutdown-timeout defaults to 5 seconds and because the >>> request still hadn't completed within 5 seconds, then the process is exited >>> anyway and allowed to shutdown. >>> >>> [Wed Jan 28 13:35:10 2015] [info] mod_wsgi (pid=29639): Aborting process >>> 'localhost:8000'. >>> [Wed Jan 28 13:35:10 2015] [info] mod_wsgi (pid=29639): Exiting process >>> 'localhost:8000'. >>> >>> Because the application never returned a response, that results in the >>> Apache child worker who was trying to talk to the daemon process seeing a >>> truncated response. >>> >>> [Wed Jan 28 13:35:10 2015] [error] [client 127.0.0.1] Truncated or >>> oversized response headers received from daemon process 'localhost:8000': >>> /tmp/mod_wsgi-localhost:8000:502/htdocs/ >>> >>> When the Apache parent process notices the daemon process has died, it >>> cleans up and starts a new one. >>> >>> [Wed Jan 28 13:35:11 2015] [info] mod_wsgi (pid=29639): Process >>> 'localhost:8000' has died, deregister and restart it. >>> [Wed Jan 28 13:35:11 2015] [info] mod_wsgi (pid=29639): Process >>> 'localhost:8000' has been deregistered and will no longer be monitored. >>> [Wed Jan 28 13:35:11 2015] [info] mod_wsgi (pid=29764): Starting process >>> 'localhost:8000' with threads=5. >>> >>> So the shutdown phase specified by shutdown-timeout is subsequent to >>> eviction-timeout. It is one last chance to shutdown during a time that no >>> new requests are accepted in case it is the constant flow of requests that >>> is preventing it, rather than one long running request. >>> >>> The shutdown-timeout should always be kept quite short because no new >>> requests will be accepted during that time. So changing it from the default >>> isn't something one would normally do. >>> >>> Graham >>> >>> On 28/01/2015, at 3:02 AM, Kent <[email protected]> wrote: >>> >>> Let me be more specific. I'm having a hard time getting this to test as I >>> expected. Here is my WSGIDaemonProcess directive: >>> >>> WSGIDaemonProcess rarch processes=3 threads=2 inactivity-timeout=1800 >>> display-name=%{GROUP} graceful-timeout=140 eviction-timeout=60 >>> python-eggs=/home/rarch/tg2env/lib/python-egg-cache >>> >>> I put a 120 sec sleep in one of the processes' requests and then SIGUSR1 >>> (Linux) all three processes. The two inactive ones immediately restart, as >>> I expect. However, the 3rd (sleeping) one is allowed to run past the 60 >>> second eviction_timeout and runs straight to the graceful_timeout before it >>> is terminated. Shouldn't it have been killed at 60 sec? >>> >>> (And then, as my previous question, how does shutdown-timeout factor into >>> all this?) >>> >>> Thanks again! >>> Kent >>> >>> >>> >>> On Tuesday, January 27, 2015 at 9:34:12 AM UTC-5, Kent wrote: >>> I think I might understand the difference between 'graceful-timeout' and >>> 'shutdown-timeout', but can you please just clarify the difference? Are >>> they additive? >>> >>> Also, will 'eviction-timeout' interact with either of those, or simply >>> override them? >>> >>> Thanks, >>> Kent >>> >>> On Monday, January 26, 2015 at 12:44:13 AM UTC-5, Graham Dumpleton wrote: >>> Want to give: >>> >>> https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.tar.gz >>> >>> a go? >>> >>> The WSGIDaemonProcess directive is 'eviction-timeout'. For mod_wsgi-express >>> the command line option is '--eviction-timeout'. >>> >>> So the terminology am using around this is that sending a signal is like >>> forcibly evicting the WSGI application, allow the process to be restarted. >>> At least this way can have an option name that is distinct enough from >>> generic 'restart' so as not to be confusing. >>> >>> Graham >>> >>> On 21/01/2015, at 11:15 PM, Kent <[email protected]> wrote: >>> >>> >>> On Tuesday, January 20, 2015 at 5:53:26 PM UTC-5, Graham Dumpleton wrote: >>> >>> On 20/01/2015, at 11:50 PM, Kent <[email protected]> wrote: >>> >>> On Sunday, January 18, 2015 at 12:43:08 AM UTC-5, Graham Dumpleton wrote: >>> There are a few possibilities here of how this could be enhanced/changed. >>> >>> The problem with maximum-requests is that it can be dangerous. People can >>> set it too low and when their site gets a big spike of traffic then the >>> processes can be restarted too quickly only adding to the load of the site >>> and causing things to slow down and hamper their ability to handle the >>> spike. This is where setting a longer amount of time for graceful-timeout >>> helps because you can set it to be quite large. The use of maximum-requests >>> can still be like using a hammer though, and one which can be applied >>> unpredictably. >>> >>> Yes, I can see that. (It may be overkill, but you could default a separate >>> minimum-lifetime parameter so only users who specifically mess with that as >>> well as maximum-requests shoot themselves in the foot, but it is starting >>> to get confusing with all the different timeouts, I'll agree there...) >>> >>> >>> The minimum-lifetime option is an interesting idea. It may have to do >>> nothing by default to avoid conflicts with existing expected behaviour. >>> >>> >>> The maximum-requests option also doesn't help in the case where you are >>> running background threads which do stuff and it is them and not the number >>> of requests coming in that dictate things like memory growth that you want >>> to counter. >>> >>> >>> True, but solving with maximum lifetime... well, actually, solving memory >>> problems with any of these mechanisms isn't measuring the heart of the >>> problem, which is RAM. I imagine there isn't a good way to measure RAM or >>> you would have added that option by now. Seems what we are truly after for >>> the majority of these isn't how many requests or how log its been up, etc, >>> but how much RAM it is taking (or perhaps, optionally, average RAM per >>> thread, instead). If my process exceeds consuming 1.5GB, then trigger a >>> graceful restart at the next appropriate convenience, being gentle to >>> existing requests. That may be arguably the most useful parameter. >>> >>> >>> The problem with calculating memory is that there isn't one cross platform >>> portable way of doing it. On Linux you have to dive into the /proc file >>> system. On MacOS X you can use C API calls. On Solaris I think you again >>> need to dive into a /proc file system but it obviously has a different file >>> structure for getting details out compared to Linux. Adding such cross >>> platform stuff in gets a bit messy. >>> >>> What I was moving towards as an extension of the monitoring stuff I am >>> doing for mod_wsgi was to have a special daemon process you can setup which >>> has access to some sort of management API. You could then create your own >>> Python script that runs in that and which using the management API can get >>> daemon process pids and then use Python psutil to get memory usage on >>> periodic basis and then you decide if process should be restarted and send >>> it a signal to stop, or management API provided which allows you to notify >>> in some way, maybe by signal, or maybe using shared memory flag, that >>> daemon process should shut down. >>> >>> >>> I figured there was something making that a pain... >>> >>> So the other option I have contemplated adding a number of times is is one >>> to periodically restart the process. The way this would work is that a >>> process restart would be done periodically based on what time was >>> specified. You could therefore say the restart interval was 3600 and it >>> would restart the process once an hour. >>> >>> The start of the time period for this would either be, when the process was >>> created, if any Python code or a WSGI script was preloaded at process start >>> time. Or, it would be from when the first request arrived if the WSGi >>> application was lazily loaded. This restart-interval could be tied to the >>> graceful-timeout option so that you can set and extended period if you want >>> to try and ensure that requests are not interrupted. >>> >>> We just wouldn't want it to die having never even served a single request, >>> so my vote would be against the birth of the process as the beginning point >>> (and, rather, at first request). >>> >>> >>> It would effectively be from first request if lazily loaded. If preloaded >>> though, as background threads could be created which do stuff and consume >>> memory over time, would then be from when process started, ie., when Python >>> code was preloaded. >>> >>> >>> But then for preloaded, processes life-cycle themselves for no reason >>> throughout inactive periods like maybe overnight. That's not the end of >>> the world, but I wonder if we're catering to the wrong design. (These are, >>> after all, webserver processes, so it seems a fair assumption that they >>> exist primarily to handle requests, else why even run under apache?) My >>> vote, for what it's worth, would still be timed from first request, but I >>> probably won't use that particular option. Either way would be useful for >>> some I'm sure. >>> >>> >>> Now we have the ability to sent the process graceful restart signal >>> (usually SIGUSR1), to force an individual process to restart. >>> >>> Right now this is tied to the graceful-timeout duration as well, which as >>> you point out, would perhaps be better off having its own time duration for >>> the notional grace period. >>> >>> Using the name restart-timeout for this could be confusing if I have a >>> restart interval option. >>> >>> >>> In my opinion, SIGUSR1 is different from the automatic parameters because >>> it was (most likely) triggered by user intervention, so that one should >>> ideally have its own parameter. If that is the case and this parameter >>> becomes dedicated to SIGUSR1, then the least ambiguous name I can think of >>> is sigusr1-timeout. >>> >>> >>> Except that it isn't guaranteed to be called SIGUSR1. Technically it could >>> be a different signal dependent on platform that Apache runs as. But then, >>> as far as I know all UNIX systems do use SIGUSR1. >>> >>> >>> In any case, they are "signals": you like signal-timeout? (Also could be >>> taken ambiguously, but maybe less so than restart-timeout?) >>> >>> I also have another type of process restart I am trying to work out how to >>> accommodate and the naming of options again complicates the problem. In >>> this case we want to introduce an artificial restart delay. >>> >>> This would be an option to combat the problem which is being caused by >>> Django 1.7 in that WSGI script file loading for Django isn't stateless. If >>> a transient problem occurs, such as the database not being ready, the >>> loading of the WSGI script file can fail. On the next request an attempt is >>> made to load it again but now Django kicks a stink because it was half way >>> setting things up last time when it failed and the setup code cannot be run >>> a second time. The result is that the process then keeps failing. >>> >>> The idea of the restart delay option therefore is to allow you to set it to >>> number of seconds, normally just 1. If set like that, if a WSGI script file >>> import fails, it will effectively block for the delay specified and when >>> over it will kill the process so the whole process is thrown away and the >>> WSGI script file can be reloaded in a fresh process. This gets rid of the >>> problem of Django initialisation not being able to be retried. >>> >>> >>> (We are using turbogears... I don't think I've seen something like that >>> happen, but rarely have seen start up anomalies.) >>> >>> A delay is needed to avoid an effective fork bomb, where a WSGI script file >>> not loading with high request throughput would cause a constant cycle of >>> processes dying and being replaced. It is possible it wouldn't be as bad as >>> I think as Apache only checks for dead processes to replace once a second, >>> but still prefer my own failsafe in case that changes. >>> >>> I am therefore totally fine with a separate graceful time period for when >>> SIGUSR1 is used, I just need to juggle these different features and come up >>> with an option naming scheme that make sense. >>> >>> How about then that I add the following new options: >>> >>> maximum-lifetime - Similar to maximum-requests in that it will cause >>> the processes to be shutdown and restarted, but in this case it will occur >>> based on the time period given as argument, measured from the first request >>> or when the WSGI script file or any other Python code was preloaded, that >>> is, in the latter case when the process was started. >>> >>> restart-timeout - Specifies a separate grace period for when the >>> process is being forcibly restarted using the graceful restart signal. If >>> restart-timeout is not specified and graceful-timeout is specified, then >>> the value of graceful-timeout is used. If neither are specified, then the >>> restart signal will be have similar to the process being sent a SIGINT. >>> >>> linger-timeout - When a WSGI script file, of other Python code is being >>> imported by mod_wsgi directly, if that fails the default is that the error >>> is ignored. For a WSGI script file reloading will be attempted on the next >>> request. But if preloading code then it will fail and merely be logged. If >>> linger-timeout is specified to a non zero value, with the value being >>> seconds, then the daemon process will instead be shutdown and restarted to >>> try and allow a successful reloading of the code to occur if it was a >>> transient issue. To avoid a fork bomb if a persistent issue, a delay will >>> be introduced based on the value of the linger-timeout option. >>> >>> How does that all sound, if it makes sense that is. :-) >>> >>> >>> >>> That sounds absolutely great! How would I get on the notification cc: of >>> the ticket or whatever so I'd be informed of progress on that? >>> >>> These days my turn around time is pretty quick so long as I am happy and >>> know what to change and how. So I just need to think a bit more about it >>> and gets some day job stuff out of the way before I can do something. >>> >>> So don't be surprised if you simply get a reply to this email within a week >>> pointing at a development version to try. >>> >>> >>> Well tons of thanks again. >>> >>> Graham >>> >>> Graham >>> >>> >>> On 17/01/2015, at 12:27 AM, Kent <[email protected]> wrote: >>> >>> Thanks again. Yes, I did take our current version from the repo because >>> you hadn't released the SIGUSR1 bit yet... I should upgrade now. >>> >>> As for the very long graceful-timeout, I was skirting around that solution >>> because I like where the setting is currently for SIGUSR1. I would like to >>> ask, "Is there a way to indicate a different graceful-timeout for handling >>> SIGUSR1 vs. maximum-requests?" but I already have the answer from the >>> release notes: "No." >>> >>> I don't know if you can see the value in distinguishing the two, but >>> maximum-requests is sort of a "standard operating mode," so it might make >>> sense for a modwsgi user to want a higher, very gentle mode of operation >>> there, whereas SIGUSR1, while beautifully more graceful than SIGKILL, still >>> "means business," so the same user may want a shorter responsive timeout >>> there (while still allowing a decent chunk of time for being graceful to >>> running requests). That is the case for me at least. Any chance you'd >>> entertain that as a feature request? >>> >>> Even if not, you've been extremely helpful, thank you! And thanks for >>> pointing me to the correct version of documentation: I thought I was >>> reading current version. >>> Kent >>> >>> P.S. I also have ideas for possible vertical URL partitioning, but >>> unfortunately, our app has much cross-over by URL, so that's why I'm down >>> this maximum-requests path... >>> >>> >>> On Friday, January 16, 2015 at 4:54:50 AM UTC-5, Graham Dumpleton wrote: >>> >>> On 16/01/2015, at 7:28 AM, Kent <[email protected]> wrote: >>> >>> I'm running 4 (a very early version of it, possibly before you officially >>> released it). We upgraded to take advantage of the amazingly-helpful >>> SIGUSR1 signaling for graceful process restarting, which we use somewhat >>> regularly to gracefully deploy software changes (minor ones which won't >>> matter if 2 processes have different versions loaded) without disrupting >>> users. Thanks a ton for that! >>> >>> SIGUSR1 support was released in version 4.1.0. >>> >>> >>> http://modwsgi.readthedocs.org/en/master/release-notes/version-4.1.0.html >>> >>> That same version has all the other stuff which was changed so long as >>> using the actual 4.1.0 is being used and you aren't still using the repo >>> from before the official release. >>> >>> If not sure, best just upgrading to latest version if you can. >>> >>> We are also multi-threading our processes (plural processes, plural >>> threads). >>> >>> Some requests could be (validly) running for very long periods of time >>> (database reporting, maybe even half hour, though that would be very >>> extreme). >>> >>> Some processes (especially those generating .pdfs, for example), hog tons >>> of RAM, as you know, so I'd like these to eventually check their RAM back >>> in, so to speak, by utilizing either inactivity-timeout or >>> maximum-requests, but always in a very gentle way, since, as I mentioned, >>> some requests might be properly running, even though for many minutes. >>> maximum-requests seems too brutal for my use-case since the threshold >>> request sends the process down the graceful-timeout/shutdown-timeout, even >>> if there are valid processes running and then SIGKILLs. My ideal vision of >>> "maximum-requests," since it is primarily for memory management, is to be >>> very gentle, sort of a "ok, now that I've hit my threshold, at my next >>> earliest convenience, I should die, but only once all my current requests >>> have ended of their own accord." >>> >>> That is where if you vertically partition those URLs out to a separate >>> daemon process group, you can simply set a very hight graceful-timeout >>> value. >>> >>> So relying on the feature: >>> >>> """ >>> 2. Add a graceful-timeout option to WSGIDaemonProcess. This option is >>> applied in a number of circumstances. >>> >>> When maximum-requests and this option are used together, when maximum >>> requests is reached, rather than immediately shutdown, potentially >>> interupting active requests if they don’t finished with shutdown timeout, >>> can specify a separate graceful shutdown period. If the all requests are >>> completed within this time frame then will shutdown immediately, otherwise >>> normal forced shutdown kicks in. In some respects this is just allowing a >>> separate shutdown timeout on cases where requests could be interrupted and >>> could avoid it if possible. >>> """ >>> >>> You could set: >>> >>> maximum-requests=20 graceful-timeout=600 >>> >>> So as soon as it hits 20 requests, it switches to a mode where it will when >>> no requests, restart. You can set that timeout as high as you want, even >>> hours, and it will not care. >>> >>> "inactivity-timeout" seems to function exactly as I want in that it seems >>> like it won't ever kill a process with a thread with an active request (at >>> least, I can't get it too even by adding a long import >>> time;time.sleep(longtime)... it doesn't seem to die until the request is >>> finished. But that's why the documentation made me nervous because it >>> implies that it could, in fact, kill a proc with an active request: "For >>> the purposes of this option, being idle means no new requests being >>> received, or no attempts by current requests to read request content or >>> generate response content for the defined period." >>> >>> The release notes for 4.1.0 say: >>> >>> """ >>> 4. The inactivity-timeout option to WSGIDaemonProcess now only results in >>> the daemon process being restarted after the idle timeout period where >>> there are no active requests. Previously it would also interrupt a long >>> running request. See the new request-timeout option for a way of >>> interrupting long running, potentially blocked requests and restarting the >>> process. >>> """ >>> >>> I'd rather have a more gentle "maximum-requests" than "inactivity-timeout" >>> because then, even on very heavy days (when RAM is most likely to choke), I >>> could gracefully turn over these processes a couple times a day, which I >>> couldn't do with "inactivity-timeout" on an extremely heavy day. >>> >>> Hope this makes sense. I'm really asking : >>> whether inactivity-timeout triggering will ever SIGKILL a process with an >>> active request, as the docs intimate >>> No from 4.1.0 onwards. >>> whether there is any way to get maximum-requests to behave more gently >>> under all circumstances >>> By setting a very very long graceful-timeout. >>> for your ideas/best advice >>> Have a good read through the release notes for 4.1.0. >>> >>> Not necessarily useful in your case, but also look at request-timeout. It >>> can act as a final fail safe for when things are stuck, but since it is >>> more of a fail safe, it doesn't make use of graceful-timeout. >>> >>> Graham >>> >>> >>> Thanks for your help! >>> >>> >>> >>> On Wednesday, January 14, 2015 at 9:48:02 PM UTC-5, Graham Dumpleton wrote: >>> >>> On 15/01/2015, at 8:32 AM, Kent <[email protected]> wrote: >>> >>> > Graham, the docs state: "For the purposes of this option, being idle >>> > means no new requests being received, or no attempts by current requests >>> > to read request content or generate response content for the defined >>> > period." >>> > >>> > This implies to me that a running request that is taking a long time >>> > could actually be killed as if it were idle (suppose it were fetching a >>> > very slow database query). Is this the case? >>> >>> This is the case for mod_wsgi prior to version 4.0. >>> >>> Things have changed in mod_wsgi 4.X. >>> >>> How long are your long running requests though? The inactivity-timeout was >>> more about restarting infrequently used applications so that memory can be >>> taken back. >>> >>> >>> > Also, I'm looking for an ultra-conservative and graceful method of >>> > recycling memory. I've read your article on url partitioning, which was >>> > useful, but sooner or later, one must rely on either inactivity-timeout >>> > or maximum-requests, is that accurate? But both these will eventually, >>> > after graceful timeout/shutdown timeout, potentially kill active >>> > requests. It is valid for our app to handle long-running reports, so I >>> > was hoping for an ultra-safe mechanism. >>> > Do you have any advice here? >>> >>> The options available in mod_wsgi 4.X are much better in this area than >>> 3.X. The changes in 4.X aren't covered in main documentation though and are >>> only described in the release notes where change was made. >>> >>> In 4.X the concept of an inactivity-timeout is now separate to the idea of >>> a request-timeout. There is also a graceful-timeout that can be applied to >>> maximum-requests and some other situations as well to allow requests to >>> finish out properly before being more brutal. One can also signal the >>> daemon processes to do a more graceful restart as well. >>> >>> You cannot totally avoid having to be brutal though and kill things else >>> you don't have a fail safe for a stuck process where all request threads >>> were blocked on back end services and were never going to recover. Use of >>> multithreading in a process also complicates the implementation of >>> request-timeout. >>> >>> Anyway, the big question is what version are you using? >>> >>> Graham >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop re >>> ... >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "modwsgi" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/modwsgi/84yzDAMFRsw/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
