I was grepping all the log files for any messages within a minute or two before the segfaults so if a request was logged anywhere I should have seen it.
I'll mention the LogLevel Debug setting to them, but there were complaints before that LogLevel Info was too noisy so I'm not sure that will fly. I'll look into the possibility of errant app threads and post back if anything turns up. Thanks very much for your help with this. On Tuesday, October 25, 2022 at 11:05:28 PM UTC-6 Graham Dumpleton wrote: > The only way I can think of that you may get a request which wasn't > logged, is if an internal Apache request was triggered via an internal > redirect from another Apache module. There still has to be an original > request, but it would be logged as different request URL to where it got > internally redirected. > > Since you are using mod_wsgi daemon mode you can likely see better > evidence of all requests being handled if turn on verbose debugging mode, > but would be quite noisy. > > LogLevel debug > WSGIVerboseDebugging On > > Graham > > On 26 Oct 2022, at 3:33 pm, stuart mcgraw <smcg...@gmail.com> wrote: > > I didn't compile mod_wsgi myself so I can't say 100% but the person who > did said so and there is a source directory on the machine named > mod_wsgi-4.9.4 with a mod_wsgi.so file whose sha1 checksum matches that of > the file in the apache modules/ directory, so I'd say I'm 99.9% sure. > > But the chances of >1Gi requests being made seem pretty small. The urls > haven't been publicized, there are only a handful of known users accessing > the urls infrequently as testers and nothing in the application would > generate requests of that magnitude. > > This may be out of scope for you, but are you aware of any (reasonably > normal) circumstances under which a mod_wsgi process could receive a > request that wasn't logged by Apache? Or perhaps I could modify the > mod_wsgi source code to print a message to a file when a request was > received (which I could then correlate with the Apache logs to answer the > question.) Because usage is very light and this is only for short term > debugging, I don't think locking or anything fancy would be needed? > > And I am still wondering about library mismatches or conflicts since > Apache, Python, mod_wsgi and C-based Python modules (eg psyocopg2) used by > the app were all built from source. It is possible that some version > mismatch there causes some memory corruption that is later manifest when > one of the mod_wsgi housekeeping threads runs? I would like if possible to > rule this out or at least put at the bottom of the list. > On Tuesday, October 25, 2022 at 8:35:07 PM UTC-6 Graham Dumpleton wrote: > >> I know you said you were using mod_wsgi/4.9.4, but are you absolutely >> sure? Apache/2.4.54 made a breaking change by changing the default for >> LimitRequestBody directive, which would cause mod_wsgi daemon process to >> crash when there were sent large request bodies over 1Gi. This was fixed in >> version 4.9.4, but am wondering whether your production system has older >> version than your development systems use and you just aren't aware of that. >> >> >> https://modwsgi.readthedocs.io/en/master/release-notes/version-4.9.4.html#bugs-fixed >> >> As to back ground threads, mod_wsgi has a couple of background threads >> which check for idle activity, deadlocks and things, but they touch so >> little they have never caused issues in the past. Beyond that, the request >> handler threads themselves should be stuck on a select loop if no requests >> are happening. >> >> On 26 Oct 2022, at 1:12 pm, stuart mcgraw <smcg...@gmail.com> wrote: >> >> Again, thanks for those suggestions. >> >> The OOM killer seems not to be an issue. I've been told there are no >> signs of it in the system logs and no signs of memory problems via >> monitoring during nomal operations. >> >> Nor did "WSGIDestroyInterpreter Off" have any effect, the segfaults are >> still occurring after that was added and Apache restarted. >> >> My understanding of how mod_wsgi works is pretty sketchy. IIUC you are >> saying that the mod_wsgi processes are sitting there, waiting on a select() >> call or the like, to receive a request from the mod_wsgi code within >> Apache; and in that state they cannot simply spontaneously crash -- it must >> be that either that the process received request from Apache (via the >> mod_wsgi module) or there is some independent thread running in the Python >> part of the mod_wsgi process (which is running my wsgi app) that is causing >> the crash? >> >> I based my claim that there were no requests coincidental with the >> segfaults based on the lack of log messages within a second or two for some >> of the segfaults. (Its a moderately busy server so of course there were >> also some close in time but for seemingly unrelated pages: eg, python, php >> or c cgi, or html.) Is it possible that the mod_wsgi processes are getting >> woken up by something that does not produce an apache access log entry? >> >> I'm still working on the python thread hypothesis (this is a production >> server so changes aren't easy.) >> On Sunday, October 23, 2022 at 2:12:02 PM UTC-6 Graham Dumpleton wrote: >> >>> How much memory do the processes use? Maybe the system OOM process >>> killer is killing the processes as they consume lots of memory and the >>> system thinks it is running low. There were some potential problems >>> introduced with Python 3.9 with how process are shutdown and that causes >>> embedded systems to fail on shutdown. >>> >>> See: >>> >>> >>> https://modwsgi.readthedocs.io/en/master/release-notes/version-4.9.1.html#features-changed >>> >>> You can try setting: >>> >>> WSGIDestroyInterpreter Off >>> >>> as mentioned in those change notes and see if it goes away. >>> >>> Other than that, if you are confident that no new requests are arriving, >>> can only suggest you work out if there are background threads running in >>> Python. >>> >>> You can do that be adding code as described in: >>> >>> >>> https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#extracting-python-stack-traces >>> >>> and triggering a dump of running threads by touching a file in the file >>> system. >>> >>> It might also be helpful if you can work out how to have the system >>> preserve core dumps from Apache so they can be used to extract a true >>> process stack trace as that may give a clue. >>> >>> Graham >>> >>> On 24 Oct 2022, at 3:51 am, stuart mcgraw <smcg...@gmail.com> wrote: >>> >>> Thanks for that suggestion. I passed it on to the site admin made and >>> he made the "application-group=%{GLOBAL}" change, but unfortunately it made >>> no difference, the segfaults are still occurring as before. Is there >>> anything else I can look at? The current configuration is: >>> >>> WSGIDaemonProcess jmwsgi processes=2 threads=10 \ >>> display-name=apache2-jmwsgi locale=en_US.UTF-8 lang=en_US.UTF-8 >>> WSGIScriptAlias /jmwsgi >>> /usr/local/apache2/jmdictdb/wsgifiles/jmdictdb.wsgi \ >>> process-group=jmwsgi application-group=%{GLOBAL} >>> >>> Would changing to "process=N threads=1" or "processes=1 threads=N" >>> provide any useful info? Apache, mod_wsgi and the other web server >>> components were all built there (ie, they are not from distro-supplied >>> packages.) Are the symptoms consistent with a mismatched library or some >>> other build configuration issue? Or conversely, maybe they make that >>> unlikely? >>> On Friday, October 21, 2022 at 11:48:51 PM UTC-6 Graham Dumpleton wrote: >>> >>>> Try changing it to: >>>> >>>> WSGIScriptAlias /jmwsgi >>>> /usr/local/apache2/jmdictdb/wsgifiles/jmdictdb.wsgi \ >>>> process-group=jmwsgi application-group=%{GLOBAL} >>>> >>>> You are possibly using a third party Python module which isn't designed >>>> to work in Python sub interpreters. That application group value forces >>>> the >>>> main Python interpreter context to be used, which can avoid problems with >>>> crashes, or thread deadlocks when such broken modules are used. >>>> >>>> >>>> https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api >>>> >>>> That option on WSGIScriptAlias has same affect as WSGIAplicationGroup >>>> but is more specific. For same reason, your use of WSGIProcessGroup is >>>> redundant as process group setting on WSGIScriptAlias takes precedence. >>>> >>>> Graham >>>> >>>> On 22 Oct 2022, at 2:35 pm, stuart mcgraw <smcg...@gmail.com> wrote: >>>> >>>> My apologies for the delayed response, I thought I had my google email >>>> forwarded to my main email account but... :-( >>>> >>>> My intent was that the processes run in daemon mode. I had missed the >>>> info about the WSGIRestrictEmbedded directive when I went through the doc, >>>> I'll ask the admin there to add that. The full configuration for wsgi is: >>>> >>>> WSGIDaemonProcess jmwsgi processes=2 threads=10 \ >>>> display-name=apache2-jmwsgi locale=en_US.UTF-8 lang=en_US.UTF-8 >>>> WSGIProcessGroup jmwsgi >>>> WSGIScriptAlias /jmwsgi >>>> /usr/local/apache2/jmdictdb/wsgifiles/jmdictdb.wsgi \ >>>> process-group=jmwsgi >>>> # Serve static files directly without using the app. >>>> Alias /jmwsgi/web/ /usr/local/apache2/jmdictdb/ >>>> <Directory /usr/local/apache2/jmdictdb> >>>> DirectoryIndex disabled >>>> Require all granted >>>> </Directory> >>>> >>>> The server has a number of virtual hosts and there were a few mod_wsgi >>>> "Loading Python" messages in the error log for one of them (for ssl) but >>>> nothing looking errorish and only a few, nowhere near the number of >>>> segfault messages: >>>> >>>> [Sat Oct 01 07:50:12.090697 2022] [wsgi:info] [pid 731154:tid >>>> 140442461062912] [remote *.*.*.*:40566] mod_wsgi (pid=731154, >>>> process='jmwsgi', application='www.edrdg.org|/jmwsgi'): Loading Python >>>> script file '/usr/local/apache2/jmdictdb/wsgifiles/jmdictdb.wsgi'. >>>> >>>> But the wsgi configuration stuff is outside all the virtutal hosts. >>>> >>>> When the server starts, there are a couple messages in the main error >>>> log file like: >>>> >>>> [Sat Oct 01 06:42:26.499086 2022] [wsgi:info] [pid 731041:tid >>>> 140442622753728] mod_wsgi (pid=731041): Starting process 'jmwsgi' with >>>> uid=33, gid=33 and threads=10. >>>> [Sat Oct 01 06:42:26.499518 2022] [wsgi:info] [pid 731039:tid >>>> 140442622753728] mod_wsgi (pid=731039): Starting process 'jmwsgi' with >>>> uid=33, gid=33 and threads=10. >>>> >>>> and these are followed/interleaved with the "Initializing Python" and >>>> "Attach interpreter" messages but after server startup the messages are >>>> limited to the sets of three I showed: "Initializing Python" and "Attach >>>> interpreter" followed sometime later by the Segmentation fault. >>>> >>>> Does any of that help? >>>> On Sunday, October 16, 2022 at 4:16:09 PM UTC-6 Graham Dumpleton wrote: >>>> >>>>> What other mod_wsgi configuration is there besides >>>>> the WSGIDaemonProcess directive? That alone only creates a mod_wsgi >>>>> daemon >>>>> process group, but does not tell mod_wsgi to use it. Thus cannot tell >>>>> whether you are using embedded mode or daemon mode. The logs are also odd >>>>> in that would expect to see other messages in there around when processes >>>>> are created if using daemon mode, plus an indication of whether a message >>>>> is being generated from an Apache child process or mod_wsgi daemon >>>>> process. >>>>> >>>>> So can you supply the other parts of the mod_wsgi configuration so can >>>>> see if properly using daemon mode or not. Also look for logs from >>>>> mod_wsgi >>>>> in any per virtual host specific error log file and not just main Apache >>>>> error log if you separate them. Finally, if you are only intending to use >>>>> mod_wsgi daemon mode, ensure you add the directive: >>>>> >>>>> WSGIRestrictEmbedded On >>>>> >>>>> outside of all VirtualHost definitions so that any attempt to >>>>> intitialise/use Python in main Apache child processes is disabled. >>>>> >>>>> Graham >>>>> >>>>> On 17 Oct 2022, at 8:07 am, stuart mcgraw <smcg...@gmail.com> wrote: >>>>> >>>>> I am author of a Flask application running under Linux/Apache mod_wsgi >>>>> that is experiencing intermittent, random segmentation faults. >>>>> >>>>> What is unusual is that the mod_wsgi process segfaults are occurring >>>>> not at startup when mod_wsgi is loaded, or at when an incoming request >>>>> accesses the app, but when the wsgi processes are just sitting there, >>>>> quiescent. >>>>> >>>>> From a user's point of view, everything looks fine, the mod_wsgi >>>>> processes and the app respond with the right results with no sign of >>>>> trouble at the client's browser. But looking at the Apache logs shows >>>>> the >>>>> wsgi processes periodically segfaulting and getting restarted with no >>>>> correlated incoming requests. They die sometimes after running for a few >>>>> minutes, sometimes after a few hours. There are no incoming requests to >>>>> the the wsgi app logged near the time of these crashes. >>>>> >>>>> For example: >>>>> [Mon May 30 22:35:43.040387 2022] [wsgi:info] [pid 2575903:tid >>>>> 139929303559104] mod_wsgi (pid=2575903): Initializing Python. >>>>> [Mon May 30 22:35:43.099053 2022] [wsgi:info] [pid 2575903:tid >>>>> 139929303559104] mod_wsgi (pid=2575903): Attach interpreter ''. >>>>> [Tue May 31 01:29:06.434000 2022] [core:notice] [pid 2876203:tid >>>>> 139929303559104] AH00052: child pid 2511562 exit signal Segmentation >>>>> fault >>>>> (11) >>>>> [Tue May 31 01:29:07.466268 2022] [wsgi:info] [pid 2605661:tid >>>>> 139929303559104] mod_wsgi (pid=2605661): Initializing Python. >>>>> [Tue May 31 01:29:07.517413 2022] [wsgi:info] [pid 2605661:tid >>>>> 139929303559104] mod_wsgi (pid=2605661): Attach interpreter ''. >>>>> [Tue May 31 04:14:59.405491 2022] [core:notice] [pid 2876203:tid >>>>> 139929303559104] AH00052: child pid 2575903 exit signal Segmentation >>>>> fault >>>>> (11) >>>>> >>>>> My wsgi app is still being tested so other than infrequent requests >>>>> generated by me and a few other people there is very little traffic to >>>>> it. >>>>> However the web server itself is handling some continuous moderate volume >>>>> of traffic to other apps including to C, Python and PHP CGI apps. >>>>> >>>>> What I know about the environment (if any other info would be useful >>>>> I'll try and dig it up): >>>>> >>>>> $ cat /etc/*release >>>>> PRETTY_NAME="Debian GNU/Linux 11 (bullseye) >>>>> >>>>> Apache, mod_wsgi, python were all built from source by the site's >>>>> administrator. >>>>> >>>>> There are (at least) two Python's on the system: >>>>> /usr/bin/python3 -- 3.9.2 >>>>> /usr/local/bin/python3 -- 3.10.1 >>>>> >>>>> Apachche/mod_wsgi is was supposedly built against python-3.10. From >>>>> the http server header: >>>>> Apache/2.4.54 (Unix) OpenSSL/1.1.1n mod_wsgi/4.9.4 Python/3.10 >>>>> PHP/7.4.23 >>>>> >>>>> The Apache .conf file uses: >>>>> WSGIDaemonProcess myapp processes=2 threads=10 \ >>>>> display-name=apache2-myapp locale=en_US.UTF-8 lang=en_US.UTF-8 >>>>> >>>>> $ /usr/local/apache2/bin/httpd -V >>>>> Server version: Apache/2.4.54 (Unix) >>>>> Server built: Oct 13 2022 00:07:38 >>>>> Server's Module Magic Number: 20120211:124 >>>>> Server loaded: APR 1.6.5, APR-UTIL 1.6.1, PCRE 10.36 2020-12-04 >>>>> Compiled using: APR 1.6.5, APR-UTIL 1.6.1, PCRE 10.36 2020-12-04 >>>>> Architecture: 64-bit >>>>> Server MPM: event >>>>> threaded: yes (fixed thread count) >>>>> forked: yes (variable process count) >>>>> Server compiled with.... >>>>> -D APR_HAS_SENDFILE >>>>> -D APR_HAS_MMAP >>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>>>> -D APR_USE_SYSVSEM_SERIALIZE >>>>> -D APR_USE_PTHREAD_SERIALIZE >>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>>>> -D APR_HAS_OTHER_CHILD >>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>>>> -D DYNAMIC_MODULE_LIMIT=256 >>>>> -D HTTPD_ROOT="/usr/local/apache2" >>>>> -D SUEXEC_BIN="/usr/local/apache2/bin/suexec" >>>>> -D DEFAULT_PIDLOG="logs/httpd.pid" >>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>>>> -D DEFAULT_ERRORLOG="logs/error_log" >>>>> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>>>> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>>>> >>>>> $ bin/httpd -M >>>>> Loaded Modules: >>>>> core_module (static) >>>>> so_module (static) >>>>> http_module (static) >>>>> mpm_event_module (static) >>>>> authz_core_module (shared) >>>>> authz_host_module (shared) >>>>> unixd_module (shared) >>>>> dir_module (shared) >>>>> access_compat_module (shared) >>>>> env_module (shared) >>>>> alias_module (shared) >>>>> log_config_module (shared) >>>>> ssl_module (shared) >>>>> mime_module (shared) >>>>> socache_shmcb_module (shared) >>>>> setenvif_module (shared) >>>>> cgid_module (shared) >>>>> userdir_module (shared) >>>>> headers_module (shared) >>>>> rewrite_module (shared) >>>>> autoindex_module (shared) >>>>> negotiation_module (shared) >>>>> dav_module (shared) >>>>> deflate_module (shared) >>>>> info_module (shared) >>>>> status_module (shared) >>>>> wsgi_module (shared) >>>>> evasive24_module (shared) >>>>> php7_module (shared) >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to modwsgi+u...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/modwsgi/e06f789f-6023-417e-8b10-1f570adc069cn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/modwsgi/e06f789f-6023-417e-8b10-1f570adc069cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to modwsgi+u...@googlegroups.com. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/093152d7-26a6-4d52-8c7b-0d4cb643fa95n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/093152d7-26a6-4d52-8c7b-0d4cb643fa95n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to modwsgi+u...@googlegroups.com. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/363f423b-5be1-4c33-8783-638c0cd72512n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/363f423b-5be1-4c33-8783-638c0cd72512n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to modwsgi+u...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/e51f7b06-b0e5-42b3-ac9c-3cc3cb89070en%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/e51f7b06-b0e5-42b3-ac9c-3cc3cb89070en%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+u...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/cecd0f01-2344-466f-9ed1-4fae73dc2762n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/cecd0f01-2344-466f-9ed1-4fae73dc2762n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/6ff197f1-c73a-42c4-b849-9418370fc9e3n%40googlegroups.com.
