Hi Julian, This conversation points to some improvements that could be made in the data structures. It is hard to picture what you are doing that your efforts would not be better rewarded by fitting your problem cleanly into Redis. Can you shed any light on specifics of your data structures?
Thanks! Jason On Apr 19, 2016 8:37 AM, "Julien Delafontaine" <[email protected]> wrote: > One problem is that the app needs to be fully loaded, i.e. models etc. I > know that in the latest versions of Django there is a hook > <https://docs.djangoproject.com/en/dev/ref/applications/#django.apps.AppConfig.ready> > that allows to run things after everything is loaded, but it did not work > as well as expected in practice. I'll try with a few seconds delay. > > I see what you mean with the mini cache server. If I don't find a simpler > way, I'll try something like this because it does exactly what I need: a > kind of Memcached without serialization. > > > Le mardi 19 avril 2016 12:48:18 UTC+2, Graham Dumpleton a écrit : >> >> One can always fire off the creation of the cache as a side affect of the >> WSGI script file being loaded. You can even do it in a background thread >> while still handling requests. So initial requests may be slow as cache >> populates, but once loaded should be good. >> >> See a problem with doing it that way? >> >> On 19 Apr 2016, at 8:45 PM, Julien Delafontaine <[email protected]> >> wrote: >> >> When a process is started, I pull blobs out of a database, put their data >> in a matrix, and keep the matrix in memory (because [de-]serialization for >> usual caches is slow) so that computations using that matrix are very >> quick. The construction of the matrix takes time, though, and can time out >> if the database is big. >> >> What I do is I trigger the first call to that controller myself (with a >> curl) so that users don't see it later and only have the quick responses. >> But if the cache gets erased, it becomes slow for them as well. >> I have a --request-timeout set to 90s, but apparently it is still not >> enough. >> >> An improvement would be to store the computed matrix in a persistent >> cache, and load that only when the app starts (takes double the amount of >> memory and still a dozen seconds to deserialize from my experience). >> >> >> Le mardi 19 avril 2016 11:33:36 UTC+2, Graham Dumpleton a écrit : >>> >>> Can you explain more about what the long running requests are doing? >>> >>> The timeout can be extended by using option like: >>> >>> —request-timeout=300 >>> >>> Would help to understand the need for long running requests and can >>> perhaps suggest a better way. >>> >>> Graham >>> >>> On 19 Apr 2016, at 7:30 PM, Julien Delafontaine <[email protected]> >>> wrote: >>> >>> I have long requests on purpose. In the same process I am building cache >>> for several elements, it can take time (at startup only), and for one item >>> it occasionally times out. So it would fit the scenario where when this one >>> times out, the process is restarted and all the previously computed data is >>> lost... This is extremely annoying :( Time to set up persistent cache, >>> maybe. >>> >>> Thanks a lot ! >>> >>> Le mardi 19 avril 2016 11:15:42 UTC+2, Graham Dumpleton a écrit : >>>> >>>> When using mod_wsgi-express it does run daemon mode. So with that >>>> configuration you should have two persistent processes. The processes >>>> should not be recycled under normal circumstances. >>>> >>>> The only way with the default configuration that processes could be >>>> recycled is if you have stuck requests and eventually trip the request >>>> timeout. For a multi thread process the process restart would kick in only >>>> when the length of all active requests (across total number of request >>>> slots) averaged 60 seconds. >>>> >>>> So if you had one stuck request only, if it was stuck for 5 minutes, >>>> then finally process would be forcibly restarted. If has two stuck requests >>>> that started at same time, would restart after 2.5 minutes. If five stuck >>>> requests in same process, then after 60 seconds. It is a weird calculation >>>> but only thing that makes half sense in multi threaded application. >>>> >>>> To work out whether forced process restarts are occurring because of >>>> the timeout, add the: >>>> >>>> —log-level info >>>> >>>> option. With this mod_wsgi will log more details about process restarts >>>> and why they were triggered. You can then look in the logs to confirm if >>>> this is what is happening. >>>> >>>> Do you know if you are seeing requests that never seem to finish? Or >>>> does your application run with very long requests on purpose? >>>> >>>> Graham >>>> >>>> On 19 Apr 2016, at 6:34 PM, Julien Delafontaine <[email protected]> >>>> wrote: >>>> >>>> I am using mod_wsgi-express: >>>> >>>> mod_wsgi-express setup-server ${baseDir}/project/wsgi.py >>>> --port=8887 --user myapp --server-root=${remoteDir}/mod_wsgi-server >>>> --processes 2 --threads 5; >>>> >>>> Then >>>> >>>> ${remoteDir}/mod_wsgi-server/apachectl restart >>>> >>>> This sets up the configuration itself, it seems. I thought >>>> mod_wsgi-express would run daemon mode by default? >>>> >>>> >>>> Le mardi 19 avril 2016 10:19:01 UTC+2, Graham Dumpleton a écrit : >>>>> >>>>> Sounds like you are using embedded mode rather than daemon mode. In >>>>> embedded mode Apache will recycle processes. >>>>> >>>>> How do you have it configured? Are you using >>>>> WSGIDaemonProcess/WSGIProcessGroup directives at all? >>>>> >>>>> Graham >>>>> >>>>> On 19 Apr 2016, at 6:12 PM, Julien Delafontaine <[email protected]> >>>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I have a multi-processes mod_wsgi application that stores some cache >>>>> data in memory. Each process naturally gets its own instance of that >>>>> cache. >>>>> Now it seems that processes after some time get killed/restarted/whatever, >>>>> so that the cache has to be reinitialized everytime this happens. How can >>>>> I >>>>> control it ? >>>>> >>>>> Ideally I'd like to start 2 Apache/mod_wsgi processes, initialize the >>>>> cache on each, and let the app run forever without needing to recompute >>>>> the >>>>> cache. Is that possible? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/modwsgi. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/modwsgi. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
