> On 25 Aug 2017, at 6:45 pm, Dave Williams <flashyd...@gmail.com> wrote: > > Thanks Graham for your prompt response. > > To answer your questions: > - Yes using mod_wsgi-express. > - onbuild: I am not wedded to this - I was just following the least line or > resistance to port my application. What precisely are you recommending and > what do I need to change (I am running docker-compose to orchestrate multiple > wsgi containers behind haproxy acting as application selector). > - The app is part of a jenkins build pipeline. The request normally takes > around 60-120 seconds. In worst case it takes 6 hours (I know! - we have > 100,000+ lines of XML across around 100 files to parse (submitted with the > request) and to write to the database as a single transaction. It is almost > impossible to split these up into small transactions and roll back if > anything should fail. It would also leave the database in a "live" but > inconsistent state until done. This has been running in full production for a > couple of years now without issue so although not architecturally nice - it > works. The dockerisation was part of providing more concurrency > - We have 1 process (default) running on each wsgi container. > - We have 5 threads (default) > - Normally expect only1 concurrent request that writes to the database (most > of the DB is write locked due to the transaction - so pointless trying more) > We try to serialise access via jenkins pipelines and reference an unoccupied > wsgi container - but expect a small number of concurrent database "read only > client" accesses (20 worst case estimate). > > Currently (bypassing the haproxy) and talking direct to a mod_wsgi container > we are getting gateway timeouts. > - I found .whiskey/server_args last night and added --include-file reference > so my /tmp/mod_wsgi-localhost:80:0/http.conf now contains Include > '/app/extra.conf' so I can add apache2 directives there. > I also tried --socket-timeout and --request-timeout as you have subsequently > recommended. > > It appears that Apache Timeout and --socket-timeout are the critical items > that need to be increased. > > Whilst that has moved me forward a bit I am subsequently hitting other > problems which result in wsgi process being abruptly restarted (with little > or nothing reported in the apache logs) when sent test data taken from my > production environment. > I am seeing "Truncated or oversized response headers received from daemon > process liocalhost:80" or simply "nothing reported" followed by "[MainThread] > Returning wsgi_app" from my wsgi bootstrap as it restarts. Client see as 500 > internal server error in this case. > I am not sure if these relate to timeouts or how to debug further. They are > very consistent in terms of time after the run starts (20 > seconds).
Very quickly on one point as is late. You want to override --request-timeout option. This defaults to 60 seconds. Even if you only had one request, with 5 threads the WSGI process will be force restarted after 300 seconds. If you don't want a timeout try --request-timeout 0. That way requests can run as long as they like and is what straight Apache/mod_wsgi behaves like. > Dave > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+unsubscr...@googlegroups.com > <mailto:modwsgi+unsubscr...@googlegroups.com>. > To post to this group, send email to modwsgi@googlegroups.com > <mailto:modwsgi@googlegroups.com>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To post to this group, send email to modwsgi@googlegroups.com. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
