There is another possible way as well, but involves more work. There is an ability in mod_wsgi-express to provide it a service script. This is a Python script which will be loaded into a daemon process of its own. This process wouldn’t handle conventional HTTP requests, but instead could set it self up as a mini cache server which you then communicate with using features of the multiprocessing module. Thus your cache resides in this process and use local request across to it to get the data needed. This process would stay persistent regardless of whether original daemon processes handling requests get restarted because of request timeouts.
If of interest I can try and find some examples where explained something similar to someone else. Graham > On 19 Apr 2016, at 8:48 PM, Graham Dumpleton <[email protected]> > wrote: > > One can always fire off the creation of the cache as a side affect of the > WSGI script file being loaded. You can even do it in a background thread > while still handling requests. So initial requests may be slow as cache > populates, but once loaded should be good. > > See a problem with doing it that way? > >> On 19 Apr 2016, at 8:45 PM, Julien Delafontaine <[email protected] >> <mailto:[email protected]>> wrote: >> >> When a process is started, I pull blobs out of a database, put their data in >> a matrix, and keep the matrix in memory (because [de-]serialization for >> usual caches is slow) so that computations using that matrix are very quick. >> The construction of the matrix takes time, though, and can time out if the >> database is big. >> >> What I do is I trigger the first call to that controller myself (with a >> curl) so that users don't see it later and only have the quick responses. >> But if the cache gets erased, it becomes slow for them as well. >> I have a --request-timeout set to 90s, but apparently it is still not enough. >> >> An improvement would be to store the computed matrix in a persistent cache, >> and load that only when the app starts (takes double the amount of memory >> and still a dozen seconds to deserialize from my experience). >> >> >> Le mardi 19 avril 2016 11:33:36 UTC+2, Graham Dumpleton a écrit : >> Can you explain more about what the long running requests are doing? >> >> The timeout can be extended by using option like: >> >> —request-timeout=300 >> >> Would help to understand the need for long running requests and can perhaps >> suggest a better way. >> >> Graham >> >>> On 19 Apr 2016, at 7:30 PM, Julien Delafontaine <mura...@ <>gmail.com >>> <http://gmail.com/>> wrote: >>> >>> I have long requests on purpose. In the same process I am building cache >>> for several elements, it can take time (at startup only), and for one item >>> it occasionally times out. So it would fit the scenario where when this one >>> times out, the process is restarted and all the previously computed data is >>> lost... This is extremely annoying :( Time to set up persistent cache, >>> maybe. >>> >>> Thanks a lot ! >>> >>> Le mardi 19 avril 2016 11:15:42 UTC+2, Graham Dumpleton a écrit : >>> When using mod_wsgi-express it does run daemon mode. So with that >>> configuration you should have two persistent processes. The processes >>> should not be recycled under normal circumstances. >>> >>> The only way with the default configuration that processes could be >>> recycled is if you have stuck requests and eventually trip the request >>> timeout. For a multi thread process the process restart would kick in only >>> when the length of all active requests (across total number of request >>> slots) averaged 60 seconds. >>> >>> So if you had one stuck request only, if it was stuck for 5 minutes, then >>> finally process would be forcibly restarted. If has two stuck requests that >>> started at same time, would restart after 2.5 minutes. If five stuck >>> requests in same process, then after 60 seconds. It is a weird calculation >>> but only thing that makes half sense in multi threaded application. >>> >>> To work out whether forced process restarts are occurring because of the >>> timeout, add the: >>> >>> —log-level info >>> >>> option. With this mod_wsgi will log more details about process restarts and >>> why they were triggered. You can then look in the logs to confirm if this >>> is what is happening. >>> >>> Do you know if you are seeing requests that never seem to finish? Or does >>> your application run with very long requests on purpose? >>> >>> Graham >>> >>>> On 19 Apr 2016, at 6:34 PM, Julien Delafontaine <mura...@ <>gmail.com >>>> <http://gmail.com/>> wrote: >>>> >>>> I am using mod_wsgi-express: >>>> >>>> mod_wsgi-express setup-server ${baseDir}/project/wsgi.py --port=8887 >>>> --user myapp --server-root=${remoteDir}/mod_wsgi-server --processes 2 >>>> --threads 5; >>>> >>>> Then >>>> >>>> ${remoteDir}/mod_wsgi-server/apachectl restart >>>> >>>> This sets up the configuration itself, it seems. I thought >>>> mod_wsgi-express would run daemon mode by default? >>>> >>>> >>>> Le mardi 19 avril 2016 10:19:01 UTC+2, Graham Dumpleton a écrit : >>>> Sounds like you are using embedded mode rather than daemon mode. In >>>> embedded mode Apache will recycle processes. >>>> >>>> How do you have it configured? Are you using >>>> WSGIDaemonProcess/WSGIProcessGroup directives at all? >>>> >>>> Graham >>>> >>>>> On 19 Apr 2016, at 6:12 PM, Julien Delafontaine <[email protected] <>> >>>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I have a multi-processes mod_wsgi application that stores some cache data >>>>> in memory. Each process naturally gets its own instance of that cache. >>>>> Now it seems that processes after some time get >>>>> killed/restarted/whatever, so that the cache has to be reinitialized >>>>> everytime this happens. How can I control it ? >>>>> >>>>> Ideally I'd like to start 2 Apache/mod_wsgi processes, initialize the >>>>> cache on each, and let the app run forever without needing to recompute >>>>> the cache. Is that possible? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>. >>>>> To post to this group, send email to mod...@ <>googlegroups.com >>>>> <http://googlegroups.com/>. >>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>> <https://groups.google.com/group/modwsgi>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] <>. >>>> To post to this group, send email to [email protected] <>. >>>> Visit this group at https://groups.google.com/group/modwsgi >>>> <https://groups.google.com/group/modwsgi>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] <>. >>> To post to this group, send email to [email protected] <>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
