On 16/01/2015, at 7:28 AM, Kent <[email protected]> wrote:
> I'm running 4 (a very early version of it, possibly before you officially
> released it). We upgraded to take advantage of the amazingly-helpful
> SIGUSR1 signaling for graceful process restarting, which we use somewhat
> regularly to gracefully deploy software changes (minor ones which won't
> matter if 2 processes have different versions loaded) without disrupting
> users. Thanks a ton for that!
SIGUSR1 support was released in version 4.1.0.
http://modwsgi.readthedocs.org/en/master/release-notes/version-4.1.0.html
That same version has all the other stuff which was changed so long as using
the actual 4.1.0 is being used and you aren't still using the repo from before
the official release.
If not sure, best just upgrading to latest version if you can.
> We are also multi-threading our processes (plural processes, plural threads).
>
> Some requests could be (validly) running for very long periods of time
> (database reporting, maybe even half hour, though that would be very extreme).
>
> Some processes (especially those generating .pdfs, for example), hog tons of
> RAM, as you know, so I'd like these to eventually check their RAM back in, so
> to speak, by utilizing either inactivity-timeout or maximum-requests, but
> always in a very gentle way, since, as I mentioned, some requests might be
> properly running, even though for many minutes. maximum-requests seems too
> brutal for my use-case since the threshold request sends the process down the
> graceful-timeout/shutdown-timeout, even if there are valid processes running
> and then SIGKILLs. My ideal vision of "maximum-requests," since it is
> primarily for memory management, is to be very gentle, sort of a "ok, now
> that I've hit my threshold, at my next earliest convenience, I should die,
> but only once all my current requests have ended of their own accord."
That is where if you vertically partition those URLs out to a separate daemon
process group, you can simply set a very hight graceful-timeout value.
So relying on the feature:
"""
2. Add a graceful-timeout option to WSGIDaemonProcess. This option is applied
in a number of circumstances.
When maximum-requests and this option are used together, when maximum requests
is reached, rather than immediately shutdown, potentially interupting active
requests if they don’t finished with shutdown timeout, can specify a separate
graceful shutdown period. If the all requests are completed within this time
frame then will shutdown immediately, otherwise normal forced shutdown kicks
in. In some respects this is just allowing a separate shutdown timeout on cases
where requests could be interrupted and could avoid it if possible.
"""
You could set:
maximum-requests=20 graceful-timeout=600
So as soon as it hits 20 requests, it switches to a mode where it will when no
requests, restart. You can set that timeout as high as you want, even hours,
and it will not care.
> "inactivity-timeout" seems to function exactly as I want in that it seems
> like it won't ever kill a process with a thread with an active request (at
> least, I can't get it too even by adding a long import
> time;time.sleep(longtime)... it doesn't seem to die until the request is
> finished. But that's why the documentation made me nervous because it
> implies that it could, in fact, kill a proc with an active request: "For the
> purposes of this option, being idle means no new requests being received, or
> no attempts by current requests to read request content or generate response
> content for the defined period."
The release notes for 4.1.0 say:
"""
4. The inactivity-timeout option to WSGIDaemonProcess now only results in the
daemon process being restarted after the idle timeout period where there are no
active requests. Previously it would also interrupt a long running request. See
the new request-timeout option for a way of interrupting long running,
potentially blocked requests and restarting the process.
"""
> I'd rather have a more gentle "maximum-requests" than "inactivity-timeout"
> because then, even on very heavy days (when RAM is most likely to choke), I
> could gracefully turn over these processes a couple times a day, which I
> couldn't do with "inactivity-timeout" on an extremely heavy day.
>
> Hope this makes sense. I'm really asking :
> whether inactivity-timeout triggering will ever SIGKILL a process with an
> active request, as the docs intimate
No from 4.1.0 onwards.
> whether there is any way to get maximum-requests to behave more gently under
> all circumstances
By setting a very very long graceful-timeout.
> for your ideas/best advice
Have a good read through the release notes for 4.1.0.
Not necessarily useful in your case, but also look at request-timeout. It can
act as a final fail safe for when things are stuck, but since it is more of a
fail safe, it doesn't make use of graceful-timeout.
Graham
> Thanks for your help!
>
>
>
> On Wednesday, January 14, 2015 at 9:48:02 PM UTC-5, Graham Dumpleton wrote:
>
> On 15/01/2015, at 8:32 AM, Kent <[email protected]> wrote:
>
> > Graham, the docs state: "For the purposes of this option, being idle means
> > no new requests being received, or no attempts by current requests to read
> > request content or generate response content for the defined period."
> >
> > This implies to me that a running request that is taking a long time could
> > actually be killed as if it were idle (suppose it were fetching a very slow
> > database query). Is this the case?
>
> This is the case for mod_wsgi prior to version 4.0.
>
> Things have changed in mod_wsgi 4.X.
>
> How long are your long running requests though? The inactivity-timeout was
> more about restarting infrequently used applications so that memory can be
> taken back.
>
>
> > Also, I'm looking for an ultra-conservative and graceful method of
> > recycling memory. I've read your article on url partitioning, which was
> > useful, but sooner or later, one must rely on either inactivity-timeout or
> > maximum-requests, is that accurate? But both these will eventually, after
> > graceful timeout/shutdown timeout, potentially kill active requests. It is
> > valid for our app to handle long-running reports, so I was hoping for an
> > ultra-safe mechanism.
> > Do you have any advice here?
>
> The options available in mod_wsgi 4.X are much better in this area than 3.X.
> The changes in 4.X aren't covered in main documentation though and are only
> described in the release notes where change was made.
>
> In 4.X the concept of an inactivity-timeout is now separate to the idea of a
> request-timeout. There is also a graceful-timeout that can be applied to
> maximum-requests and some other situations as well to allow requests to
> finish out properly before being more brutal. One can also signal the daemon
> processes to do a more graceful restart as well.
>
> You cannot totally avoid having to be brutal though and kill things else you
> don't have a fail safe for a stuck process where all request threads were
> blocked on back end services and were never going to recover. Use of
> multithreading in a process also complicates the implementation of
> request-timeout.
>
> Anyway, the big question is what version are you using?
>
> Graham
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
