If proc filesystem isn’t available will always return None. I think the proxy system should always be available, unless is very special Linux build. But then that means always return None and not transiently.
> On 8 Apr 2016, at 6:33 AM, Kent Bower <[email protected]> wrote: > > It's Linux, mod_wsgi version 4.4.22 > > I can go to 4.5.1, but if the proc file system isn't available for some > reason (I assume transiently??), will 4.5.1 still return None? > > Kent > > On Thu, Apr 7, 2016 at 3:51 PM, Graham Dumpleton <[email protected] > <mailto:[email protected]>> wrote: > When you are not using Linux or MacOS X. Or when the Linux system you are > using doesn’t provide proc file system for some reason. > > Also, randomly if using the develop version from the Git repository before > 4.5.0 was released, plus if you are using 4.5.0. :-) > > The randomly bug should be fixed by using 4.5.1. > > So I have now released this code, but detected the same issue about five > minutes after releasing 4.5.0, as was the first time I had tried on Linux. > Thus 4.5.1 was released to fix it. > > Try again with the latest version. > > Graham > > >> On 8 Apr 2016, at 12:27 AM, Kent Bower <[email protected] >> <mailto:[email protected]>> wrote: >> >> Graham, >> >> Under what circumstances might mod_wsgi.process_metrics() return None? >> >> Exception in thread Thread-1: >> Traceback (most recent call last): >> File "/usr/local/lib/python2.6/threading.py", line 525, in >> __bootstrap_inner >> self.run() >> File "/usr/local/lib/python2.6/threading.py", line 477, in run >> self.__target(*self.__args, **self.__kwargs) >> File "/home/rarch/trunk/src/appserver/wsgi-config/memory_monitor.py", line >> 41, in monitor >> megs = metrics['memory_rss']/1048576 >> TypeError: 'NoneType' object is unsubscriptable >> >> I've only seen this once in apache log file, some strange timing?? >> >> Kent >> >> >> On Mon, Mar 28, 2016 at 8:42 AM, Kent Bower <[email protected] >> <mailto:[email protected]>> wrote: >> (Graham, your suggestions and recipe for a memory monitoring thread are >> working beautifully. Thanks again.) >> >> On Mon, Mar 21, 2016 at 6:30 PM, Graham Dumpleton >> <[email protected] <mailto:[email protected]>> wrote: >> >>> On 22 Mar 2016, at 4:01 AM, Kent Bower <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> In your recipe for a background monitoring thread watching memory >>> consumption, after issuing the SIGUSR1, I'd probably just want the thread >>> to exit instead of sleeping... do I just do "sys.exit()" to safely >>> accomplish that? >> >> The code isn’t just sleeping. It waits on a queue object which has something >> placed on it when mod_wsgi is shutting down the process via atexit callback. >> When the thread gets that it will exit cleanly, with the main thread waiting >> on it to exit to ensure it isn’t running. >> >> If you just call sys.exit() that results in a SystemExit exception being >> raised which causes the thread to exit but leaves an exception in the error >> logs. >> >> The use of the queue is better as it ensures that threads are shutdown >> properly when process is shutting down, else you risk that the thread could >> try and run while interpreter is being destroyed, causing Python to crash >> the process. >> >>> Also, regarding my observations of paster returning garbage-collected >>> memory to the OS, was I just getting lucky while monitoring (the memory was >>> at the very top of the allocated memory)? This is a universal python issue? >> >> It is a universal issue with any programs running on a UNIX system. >> >> You may want to Google up some articles on how memory allocation in UNIX as >> well as in Python works. >> >>> >>> Again, thanks for all your help! >>> >>> On Sat, Mar 19, 2016 at 11:22 PM, Graham Dumpleton >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>>> On 20 Mar 2016, at 1:10 AM, Kent Bower <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Thanks Graham, few more items inline... >>>> >>>> On Sat, Mar 19, 2016 at 1:24 AM, Graham Dumpleton >>>> <[email protected] <mailto:[email protected]>> wrote: >>>> >>>>> On 17 Mar 2016, at 11:28 PM, Kent Bower <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> My answers are below, but before you peek, Graham, note that you and I >>>>> have been through this memory discussion before & I've read the vertical >>>>> partitioning article and use inactivity-timeout, "WSGIRestrictEmbedded >>>>> On", considered maximum-requests, etc. >>>>> >>>>> After years of this, I'm resigned to the fact that python is memory >>>>> hungry, especially built on many of these web-stack and database >>>>> libraries, etc. I'm Ok with that. I'm fine with a high-water RAM mark >>>>> imposed by running under Apache, mostly. But, dang, it sure would be >>>>> great if the 1 or 2% of requests that really (and legitimately) hog a ton >>>>> of RAM, like, say 500MB extra, didn't keep it when done. I may revisit >>>>> vertical partitioning again, but last time I did I think I found that the >>>>> 1 or 2% in my case generally won't be divisible by url. In most cases I >>>>> wouldn't know whether the particular request is going to need lots of RAM >>>>> until after the database queries return (which is far too late for >>>>> vertical partitioning to be useful). >>>>> >>>>> So I was mostly just curious about the status of nginx running wsgi, >>>>> which doesn't solve python's memory piggishness, but would at least >>>>> relinquish the extra RAM once python garbage collected. >>>> >>>> Where have you got the idea that using nginx would result in memory being >>>> released back to the OS once garbage collected? It isn’t able to do that. >>>> >>>> The situations are very narrow as to when a process is able to give back >>>> memory to the operating system. It can only be done when the now free >>>> memory was at top of allocated memory. This generally only happens for >>>> large block allocations and not in normal circumstances for a running >>>> Python application. >>>> >>>> >>>> At this point I'm not sure where I got that idea, but I'm surprised at >>>> this. For example, my previous observations of paster running wsgi were >>>> that it is quite faithful at returning free memory to the OS. Was I just >>>> getting lucky, or would paster be different for some reason? >>>> >>>> In any case, if nginx won't solve that, then I can't see any reason to >>>> even consider it over apache/mod_wsgi. Thank you for answering that. >>>> >>>> >>>>> (Have you considered a max-memory parameter to mod_wsgi that would >>>>> gracefully stop taking requests and shutdown after the threshold is >>>>> reached for platforms that would support it? I recall -- maybe >>>>> incorrectly -- you saying on Windows or certain platforms you wouldn't be >>>>> able to support that. What about the platforms that could support it? >>>>> It seems to me to be the very best way mod_wsgi could approach this >>>>> Apache RAM nuance, so seems like it would be tremendously useful for the >>>>> platforms that could support it.) >>>> >>>> You can do this yourself rather easily with more recent mod_wsgi version. >>>> >>>> If you create a background thread from a WSGI script file, in similar way >>>> as monitor for code changes does in: >>>> >>>> >>>> http://modwsgi.readthedocs.org/en/develop/user-guides/debugging-techniques.html#extracting-python-stack-traces >>>> >>>> <http://modwsgi.readthedocs.org/en/develop/user-guides/debugging-techniques.html#extracting-python-stack-traces> >>>> >>>> but instead of looking for code changes, inside the main loop of the >>>> background thread do: >>>> >>>> import os >>>> import mod_wsgi >>>> >>>> metrics = mod_wsgi.process_metrics() >>>> >>>> if metrics[‘memory_rss’] > MYMEMORYTHRESHOLD: >>>> os.kill(os.getpid(), signal.SIGUSR1) >>>> >>>> So mod_wsgi provides the way of determining the amount of memory without >>>> resorting to importing psutil, which is quite fat in itself, but how you >>>> use it is up to you. >>>> >>>> >>>> Right, that's an idea; (could even be a shell script that takes this >>>> approach, I suppose, but I like your recipe.) >>>> >>>> Unfortunately, I don't want to automate bits that can feasibly clobber >>>> blocked sessions. SIGUSR1, after graceful-timeout & shutdown-timeout, can >>>> result in ungraceful killing. Our application shares a database with an >>>> old legacy application which was poorly written to hold transactions while >>>> waiting on user input (this was apparently common two decades ago). So, >>>> unfortunately, it isn't terribly uncommon that our application is blocked >>>> at the database level waiting for someone using the legacy application who >>>> has a record(s) locked and may not even be at their desk or may have gone >>>> to lunch. Sometimes our client's IT staff has to hunt down these people >>>> or decide to kill their database session. In any case, from a >>>> professional point of view, our application should be the responsible one >>>> and wait patiently, allowing our client's IT staff the choice of how to >>>> handle those cases. So, while the likelihood is pretty low, even with >>>> graceful-timeout & shutdown-timeout set at a very high value like 5 >>>> minutes, I still run the risk of killing legitimate sessions with SIGUSR1. >>>> (I've brought this up before and you didn't agree with my gripe and I do >>>> understand why, but in my use case, I don't feel I can automate that route >>>> responsibly.... we do use SIGUSR1 manually sometimes, when we can monitor >>>> and react to cases where a session is blocked at the database level.) >>> >>> If we have discussed it previously, then I may not have anything more to >>> add. >>> >>> Did I previously suggest offloading this memory consuming tasks behind a >>> job queue run under Celery or something else? That way they are out of the >>> web server processes at least. >>> >>>> inactivity-timeout doesn't present this concern: it won't ever kill >>>> anything, just silently restarts like a good boy when inactive. I've >>>> recently reconsidered dropping that way down from 30 minutes. (When I >>>> first implemented this, it was just to reclaim RAM at the end of the day, >>>> so that's why it is 30 minutes. I didn't like the idea of churning new >>>> processes during busy periods, but I've been thinking 1 or 2 minutes may >>>> be quite reasonable.) >>>> >>>> If I could signal processes to shutdown at their next opportunity (meaning >>>> the next time they are handling no requests, like inactivity-timeout), >>>> that would solve many issues in this regard for me because I could signal >>>> these processes when their RAM consumption is high and let them restart >>>> when "convenient," being the ultimate in gracefulness. SIGUSR2 could mean >>>> "the next time you get are completely idle," while SIGUSR1 continues to >>>> mean "initiate shutdown now.” >>> >>> That is what SIGUSR1 does it you set graceful-timeout large enough. It is >>> SIGINT or SIGTERM which is effectively initiate shutdown now. So shouldn’t >>> be a need to have a SIGUSR2 as SIGUSR1 should already do what you are >>> hoping for with a reasonable setting of graceful-timeout. >>> >>>> >>>> Do note that if using SIGUSR1 to restart the current process (which should >>>> only be done for deamon mode), you should also set graceful-timeout option >>>> to WSGIDaemonProcess if you have long running requests. It is the maximum >>>> time process will wait to shutdown while still waiting for requests when >>>> doing a SIGUSR2 graceful shutdown of process, before going into forced >>>> shutdown mode where no requests will be accepted and requests can be >>>> interrupted. >>>> >>>>> Here >>>>> (http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html >>>>> >>>>> <http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html>) >>>>> you discuss nginx's tendency to block requests that may otherwise be >>>>> executing in a different process, depending on timing, etc. Is this >>>>> issue still the same (I thought I read a hint somewhere that there may be >>>>> a workaround for that), so I ask. >>>> >>>> That was related to someones attempt to embedded a Python interpreter >>>> inside of nginx processes themselves. That project died a long time ago. >>>> No one embeds Python interpreters inside of nginx processes. It was a >>>> flawed design. >>>> >>>> I don’t what you are reading to get all these strange ideas. :-) >>>> >>>> >>>> Google, I suppose ;) That's why I finally asked you when I couldn't find >>>> anything more about it via Google. >>>> >>>> >>>> >>>>> And so I wanted your opinion on nginx... >>>>> >>>>> ==== >>>>> Here is what you asked for if it can still be useful. >>>>> >>>>> I'm on mod_wsgi-4.4.6 and the particular server that prompted me this >>>>> time is running Apache 2.4 (prefork), though some of our clients use 2.2 >>>>> (prefork). >>>>> >>>>> Our typical wsgi conf setting is something like this, though threads and >>>>> processes varies depending on server size: >>>>> >>>>> LoadModule wsgi_module modules/mod_wsgi.so >>>>> WSGIPythonHome /home/rarch/tg2env >>>>> # see http://code.google.com/p/modwsgi/issues/detail?id=196#c10 >>>>> <http://code.google.com/p/modwsgi/issues/detail?id=196#c10> concerning >>>>> timeouts >>>>> WSGIDaemonProcess rarch processes=20 threads=14 inactivity-timeout=1800 >>>>> display-name=%{GROUP} graceful-timeout=5 >>>>> python-eggs=/home/rarch/tg2env/lib/python-egg-cache >>>> >>>> Is your web server really going to be idle for 30 minutes? I can’t see how >>>> that would have been doing anything. >>>> >>>> Also, in mod_wsgi 4.x when inactivity-timeout kicks in has changed. >>>> >>>> It used to apply when there were active requests and they were blocked, as >>>> well as when no requests were running. >>>> >>>> Now it only applies to case where there are no requests. >>>> >>>> The case for running but blocked requests is now handled by >>>> request-timeout. >>>> >>>> You may be better of setting request-timeout now to be a more reasonable >>>> value for your expected longest request, but set inactivity-timeout to >>>> something much shorter. >>>> >>>> So suggest you play with that. >>>> >>>> Also, are you request handles I/O or CPU intensive and how many requests? >>>> >>>> Such a high number of processes and threads always screams to me that half >>>> the performance problems are due to setting these too [HIGH], invoking >>>> pathological OS process swapping issues and Python GIL issues. >>>> >>>> >>>> >>>> Yes, the requests are I/O intensive (that is, database intensive, which >>>> adds a huge overhead to our typical request). Often requests finish in >>>> under a second or two, but they also can take many seconds (not terrible >>>> for the user, but sometimes they do a lot of processing with many trips to >>>> the database). >>>> We have several clients (companies), so the number of requests varies >>>> widely, but can get pretty heavy on busy days (like black friday, since >>>> they are in retail). We've played with those numbers quite a bit and >>>> without high numbers like that, responsiveness suffers because we backlog >>>> due to requests often taking several seconds. >>>> >>>> Thanks for all your input, you've been tremendously helpful! >>>> Kent >>>> >>>> >>>> >>>>> WSGIProcessGroup rarch >>>>> WSGISocketPrefix run/wsgi >>>>> WSGIRestrictStdout Off >>>>> WSGIRestrictStdin On >>>>> # Memory tweak. >>>>> http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html >>>>> <http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html> >>>>> WSGIRestrictEmbedded On >>>>> WSGIPassAuthorization On >>>>> >>>>> # we'll make the /tg/ directory resolve as the wsgi script >>>>> WSGIScriptAlias /tg >>>>> /home/rarch/trunk/src/appserver/wsgi-config/wsgi-deployment.py >>>>> process-group=rarch application-group=%{GLOBAL} >>>>> WSGIScriptAlias /debug/tg >>>>> /home/rarch/trunk/src/appserver/wsgi-config/wsgi-deployment.py >>>>> process-group=rarch application-group=%{GLOBAL} >>>>> >>>>> MaxRequestsPerChild 0 >>>>> <IfModule prefork.c> >>>>> MaxClients 308 >>>>> ServerLimit 308 >>>>> </IfModule> >>>>> <IfModule worker.c> >>>>> ThreadsPerChild 25 >>>>> MaxClients 400 >>>>> ServerLimit 16 >>>>> </IfModule> >>>>> >>>>> >>>>> Thanks for all your help and for excellent software! >>>>> Kent >>>>> >>>>> >>>>> On Wed, Mar 16, 2016 at 7:27 PM, Graham Dumpleton >>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>> On the question of whether nginx will solve this problem, I can’t see how. >>>>> >>>>> When one talks about nginx and Python web applications, it is only as a >>>>> proxy for HTTP requests to some backend WSGI server. The Python web >>>>> application doesn’t run in nginx itself. So memory issues and how to deal >>>>> with them are the provence of the WSGI server used, whatever that is and >>>>> not nginx. >>>>> >>>>> Anyway, answer the questions below and can start with that. >>>>> >>>>> You really want to be using recent mod_wsgi version and not Apache 2.2. >>>>> >>>>> Apache 2.2 design has various issues and bad configuration defaults which >>>>> means it can gobble up more memory than you want. Recent mod_wsgi >>>>> versions have workarounds for Apache 2.2 issues and are much better at >>>>> eliminating those Apache 2.2 issues. Recent mod_wsgi versions also have >>>>> fixes for memory usage problems in some corner cases. As far as what I >>>>> mean by recent, I recommend 4.4.12 or later. The most recent version is >>>>> 4.4.21. If you are stuck with 3.4 or 3.5 from your Linux distro that is >>>>> not good and that may increase problems. >>>>> >>>>> So long as got recent mod_wsgi version then can look at using vertical >>>>> partitioning to farm out memory hungry request handlers to their own >>>>> daemon process group and better configure those to handle that and >>>>> recycle processes based on activity or, memory usage. A blog post related >>>>> to that is: >>>>> >>>>> * >>>>> http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html >>>>> <http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html> >>>>> >>>>> Graham >>>>> >>>>>> On 17 Mar 2016, at 7:15 AM, Graham Dumpleton <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> What version of mod_wsgi and Apache are you using? >>>>>> >>>>>> Are you stuck with old versions of both? >>>>>> >>>>>> For memory tracking there are API calls mod_wsgi provides in recent >>>>>> versions for getting memory usage which can be used as part of scheme to >>>>>> trigger a process restart. You can’t use sys.exit(), but can use signals >>>>>> to trigger a clean shutdown of a process. Again better to have recent >>>>>> mod_wsgi versions as can then also set up some graceful timeout options >>>>>> for signal induced restart. >>>>>> >>>>>> Also, what is your mod_wsgi configuration so can make sure doing all the >>>>>> typical things one would do to limit memory usage, or quarantine >>>>>> particular handlers which are memory hungry? >>>>>> >>>>>> Graham >>>>>> >>>>>>> On 17 Mar 2016, at 4:29 AM, Kent Bower <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Interesting idea.. yes, we are using multiple threads and also other >>>>>>> stack frameworks, so that's not straightforward, but worth thinking >>>>>>> about... not sure how to approach that with the other threads. Thank >>>>>>> you Bill. >>>>>>> >>>>>>> On Wed, Mar 16, 2016 at 1:11 PM, Bill Freeman <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> I don't know about nginx, but one possibility, if the large memory >>>>>>> requests are infrequent, is to detect when you have completed one and >>>>>>> trigger the exit/reload of the daemon process (calling sys.exit() is >>>>>>> not the way, since there could be other threads in the middle of >>>>>>> something, unless you run one thread per process). >>>>>>> >>>>>>> On Wed, Mar 16, 2016 at 7:50 AM, Kent <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> I'm looking for a very brief high-level pros vs. cons of wsgi under >>>>>>> apache vs. under nginx and then to be pointed to more details I can >>>>>>> study myself (or at least the latter). >>>>>>> >>>>>>> Our application occasionally allows requests that consume a large >>>>>>> amount of RAM (no obvious way around that, they are valid requests) and >>>>>>> occasionally this causes problems since we can't reclaim the RAM >>>>>>> readily from apache. (We already have tweaked with and do use >>>>>>> "inactivity-timeout". This helps, but still now and then we hit >>>>>>> problems where we run into swapping to disk.) >>>>>>> >>>>>>> I'm wondering if nginx may solve this problem. I've read much of what >>>>>>> you (Graham) have had to say about the memory strategies with apache >>>>>>> and mod_wsgi, but wonder what your opinion of nginx is and where you've >>>>>>> already discussed this. I've read articles I could find you've written >>>>>>> on nginx, such as "Blocking requests and nginx version of mod_wsgi," >>>>>>> but wonder if the same weaknesses are still applicable today, 7 years >>>>>>> later? >>>>>>> >>>>>>> >>>>>>> Thank you very much in advance! >>>>>>> Kent >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "modwsgi" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To post to this group, send email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>>>> <https://groups.google.com/group/modwsgi>. >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in the >>>>>>> Google Groups "modwsgi" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe >>>>>>> <https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe>. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To post to this group, send email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>>>> <https://groups.google.com/group/modwsgi>. >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "modwsgi" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To post to this group, send email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>>>> <https://groups.google.com/group/modwsgi>. >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "modwsgi" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe >>>>> <https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe>. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected] >>>>> <mailto:[email protected]>. >>>>> To post to this group, send email to [email protected] >>>>> <mailto:[email protected]>. >>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>> <https://groups.google.com/group/modwsgi>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] >>>>> <mailto:[email protected]>. >>>>> To post to this group, send email to [email protected] >>>>> <mailto:[email protected]>. >>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>> <https://groups.google.com/group/modwsgi>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "modwsgi" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe >>>> <https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe>. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] >>>> <mailto:[email protected]>. >>>> To post to this group, send email to [email protected] >>>> <mailto:[email protected]>. >>>> Visit this group at https://groups.google.com/group/modwsgi >>>> <https://groups.google.com/group/modwsgi>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] >>>> <mailto:[email protected]>. >>>> To post to this group, send email to [email protected] >>>> <mailto:[email protected]>. >>>> Visit this group at https://groups.google.com/group/modwsgi >>>> <https://groups.google.com/group/modwsgi>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe >>> <https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to [email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to [email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "modwsgi" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe >> <https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "modwsgi" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe > <https://groups.google.com/d/topic/modwsgi/wyo2bJP0Cfc/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
