Do you have to use prefork MPM?
[Tue Mar 27 12:16:03.908422 2018] [http2:warn] [pid 11478] AH10034: The mpm
module (prefork.c) is not supported by mod_http2. The mpm determines how things
are processed in your server. HTTP/2 has more demands in this regard and the
currently selected mpm will just not do. This is an advisory warning. Your
server will continue to work, but the HTTP/2 protocol will be inactive.
Can you use event MPM?
Anyway, the line:
[Tue Mar 27 12:16:30.941339 2018] [core:notice] [pid 11478] AH00052: child pid
11480 exit signal Segmentation fault (11)
confirms the cause is that the mod_wsgi daemon process is crashing while
handling the request.
Because you are using mod_ssl and using a self compiled Python version, one
problem may be that mod_ssl was compiled and linked against a different version
of the SSL libraries, than what Python was compiled against. This will cause
issues as that loaded by mod_ssl will be loaded first.
What do you get if you run 'ldd' against the mod_ssl.so file?
What do you get if you run 'ldd' against the _ssl.so in the Python installation?
Other than that guess, is hard to pin down why it might crash. You can do:
http://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html#debugging-crashes-with-gdb
<http://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html#debugging-crashes-with-gdb>
but sounds like you aren't going to have gdb on the system and probably aren't
going to be able to run it as root anyway.
BTW, have you at least tried testing with a WSGI hello world program rather
than your full application? IOW, have you got any WSGI application working with
that Apache/Python setup?
Graham
> On 27 Mar 2018, at 11:57 pm, Eric Sears <[email protected]> wrote:
>
> Graham,
>
> Requested info below. I reset the configuration to how it is working on our
> 2.2 install.
>
> httpd.conf
>
>
> LoadModule wsgi_module
> "/usr/local/lib/python2.7/site-packages/mod_wsgi-4.6.3-py2.7-linux-x86_64.egg/mod_wsgi/server/mod_wsgi-py27.so"
>
> <VirtualHost 192.168.1.2:443>
> DocumentRoot "/opt/rh/httpd24/root/var/www/html/sitea/project_name"
> ServerName site.name.here
> Alias /static/ /opt/rh/httpd24/root/var/html/sitea/static/
>
> WSGIDaemonProcess project_name
> python-path=/opt/rh/httpd24/root/var/www/html/sitea
> WSGIProcessGroup project_name
> WSGIScriptAlias / /opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py
> WSGIApplicationGroup %{GLOBAL}
>
> DocumentRoot /opt/rh/httpd24/root/var/www/html/sitea
>
> ----
>
> error_log
>
> [Tue Mar 27 12:16:03.658057 2018] [core:notice] [pid 11475] SELinux policy
> enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
> [Tue Mar 27 12:16:03.733379 2018] [suexec:notice] [pid 11475] AH01232: suEXEC
> mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
> [Tue Mar 27 12:16:03.904147 2018] [ssl:notice] [pid 11478] AH01884: Operating
> in SSL FIPS mode
> [Tue Mar 27 12:16:03.904401 2018] [socache_shmcb:info] [pid 11478] AH00830:
> Shared memory socache initialised
> [Tue Mar 27 12:16:03.904417 2018] [ssl:info] [pid 11478] AH01887: Init:
> Initializing (virtual) servers for SSL
> [Tue Mar 27 12:16:03.908292 2018] [ssl:info] [pid 11478] AH01876:
> mod_ssl/2.4.27 compiled against Server: Apache/2.4.27, Library: OpenSSL/1.0.1e
> [Tue Mar 27 12:16:03.908338 2018] [http2:info] [pid 11478] AH03090: mod_http2
> (v1.10.7, feats=CHPRIO+SHA256, nghttp2 1.7.1), initializing...
> [Tue Mar 27 12:16:03.908422 2018] [http2:warn] [pid 11478] AH10034: The mpm
> module (prefork.c) is not supported by mod_http2. The mpm determines how
> things are processed in your server. HTTP/2 has more demands in this regard
> and the currently selected mpm will just not do. This is an advisory warning.
> Your server will continue to work, but the HTTP/2 protocol will be inactive.
> [Tue Mar 27 12:16:03.911169 2018] [wsgi:info] [pid 11480] mod_wsgi
> (pid=11480): Starting process 'project_name' with uid=503, gid=503 and
> threads=15.
> [Tue Mar 27 12:16:03.911908 2018] [wsgi:info] [pid 11480] mod_wsgi
> (pid=11480): Initializing Python.
> [Tue Mar 27 12:16:03.912771 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Initializing Python.
> [Tue Mar 27 12:16:03.913749 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Initializing Python.
> [Tue Mar 27 12:16:03.914188 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Initializing Python.
> [Tue Mar 27 12:16:03.914551 2018] [mpm_prefork:notice] [pid 11478] AH00163:
> Apache/2.4.27 (Red Hat) OpenSSL/1.0.1e-fips mod_wsgi/4.6.3 Python/2.7
> configured -- resuming normal operations
> [Tue Mar 27 12:16:03.914572 2018] [mpm_prefork:info] [pid 11478] AH00164:
> Server built: Sep 20 2017 09:57:20
> [Tue Mar 27 12:16:03.914588 2018] [core:notice] [pid 11478] AH00094: Command
> line: '/opt/rh/httpd24/root/usr/sbin/httpd'
> [Tue Mar 27 12:16:03.914797 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Initializing Python.
> [Tue Mar 27 12:16:03.915656 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Initializing Python.
> [Tue Mar 27 12:16:03.930968 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Attach interpreter ''.
> [Tue Mar 27 12:16:03.932373 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Attach interpreter ''.
> [Tue Mar 27 12:16:03.932402 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Attach interpreter ''.
> [Tue Mar 27 12:16:03.932611 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Attach interpreter ''.
> [Tue Mar 27 12:16:03.932627 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:03.933738 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Attach interpreter ''.
> [Tue Mar 27 12:16:03.934065 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:03.934171 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:03.934290 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:03.935374 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:25.937362 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Initializing Python.
> [Tue Mar 27 12:16:25.949578 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Attach interpreter ''.
> [Tue Mar 27 12:16:25.950553 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:30.941339 2018] [core:notice] [pid 11478] AH00052: child
> pid 11480 exit signal Segmentation fault (11)
> [Tue Mar 27 12:16:30.941397 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11480): Process 'project_name' has died, deregister and restart it.
> [Tue Mar 27 12:16:30.941411 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11480): Process 'project_name' terminated by signal 11
> [Tue Mar 27 12:16:30.941423 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11480): Process 'project_name' has been deregistered and will no longer
> be monitored.
> [Tue Mar 27 12:16:30.942513 2018] [wsgi:info] [pid 11570] mod_wsgi
> (pid=11570): Starting process 'project_name' with uid=503, gid=503 and
> threads=15.
> [Tue Mar 27 12:16:30.943062 2018] [wsgi:info] [pid 11570] mod_wsgi
> (pid=11570): Initializing Python.
> [Tue Mar 27 12:16:48.959264 2018] [core:notice] [pid 11478] AH00052: child
> pid 11570 exit signal Segmentation fault (11)
> [Tue Mar 27 12:16:48.959306 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11570): Process 'project_name' has died, deregister and restart it.
> [Tue Mar 27 12:16:48.959317 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11570): Process 'project_name' terminated by signal 11
> [Tue Mar 27 12:16:48.959327 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11570): Process 'project_name' has been deregistered and will no longer
> be monitored.
> [Tue Mar 27 12:16:48.960440 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Starting process 'project_name' with uid=503, gid=503 and
> threads=15.
> [Tue Mar 27 12:16:48.960991 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Initializing Python.
> [Tue Mar 27 12:17:14.221919 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11607): Process 'project_name' to be deregistered, as server is
> restarting or being shutdown.
> [Tue Mar 27 12:17:14.221969 2018] [wsgi:info] [pid 11478] mod_wsgi
> (pid=11607): Process 'project_name' has been deregistered and will no longer
> be monitored.
> [Tue Mar 27 12:17:14.222039 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Destroying interpreters.
> [Tue Mar 27 12:17:14.222065 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.222112 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Destroying interpreters.
> [Tue Mar 27 12:17:14.222139 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.222139 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Destroying interpreters.
> [Tue Mar 27 12:17:14.222173 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.222179 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Destroying interpreters.
> [Tue Mar 27 12:17:14.222208 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.223406 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Destroying interpreters.
> [Tue Mar 27 12:17:14.223435 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.224795 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Destroying interpreters.
> [Tue Mar 27 12:17:14.224825 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.228216 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Terminating Python.
> [Tue Mar 27 12:17:14.228215 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Terminating Python.
> [Tue Mar 27 12:17:14.228416 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Terminating Python.
> [Tue Mar 27 12:17:14.228487 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Terminating Python.
> [Tue Mar 27 12:17:14.228864 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Terminating Python.
> [Tue Mar 27 12:17:14.230253 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Terminating Python.
> [Tue Mar 27 12:17:14.230969 2018] [wsgi:info] [pid 11492] mod_wsgi
> (pid=11492): Python has shutdown.
> [Tue Mar 27 12:17:14.230975 2018] [wsgi:info] [pid 11550] mod_wsgi
> (pid=11550): Python has shutdown.
> [Tue Mar 27 12:17:14.231121 2018] [wsgi:info] [pid 11497] mod_wsgi
> (pid=11497): Python has shutdown.
> [Tue Mar 27 12:17:14.231637 2018] [wsgi:info] [pid 11487] mod_wsgi
> (pid=11487): Python has shutdown.
> [Tue Mar 27 12:17:14.231642 2018] [wsgi:info] [pid 11481] mod_wsgi
> (pid=11481): Python has shutdown.
> [Tue Mar 27 12:17:14.233003 2018] [wsgi:info] [pid 11482] mod_wsgi
> (pid=11482): Python has shutdown.
> [Tue Mar 27 12:17:14.239766 2018] [core:info] [pid 11478] AH00096: removed
> PID file /opt/rh/httpd24/root/var/run/httpd/httpd.pid (pid=11478)
> [Tue Mar 27 12:17:14.239786 2018] [mpm_prefork:notice] [pid 11478] AH00169:
> caught SIGTERM, shutting down
>
> ---
>
> ssl_error_log
>
> SSL INFO REDACTED
> [Tue Mar 27 12:16:03.930991 2018] [wsgi:info] [pid 11480] mod_wsgi
> (pid=11480): Attach interpreter ''.
> [Tue Mar 27 12:16:03.931207 2018] [wsgi:info] [pid 11480] mod_wsgi
> (pid=11480): Adding '/opt/rh/httpd24/root/var/www/html/sitea' to path.
> [Tue Mar 27 12:16:03.933442 2018] [wsgi:info] [pid 11480] mod_wsgi
> (pid=11480): Imported 'mod_wsgi'.
>
> [Tue Mar 27 12:16:30.010940 2018] [wsgi:info] [pid 11480] [remote
> 192.168.1.100:21802] mod_wsgi (pid=11480, process='project_name',
> application=''): Loading Python script file
> '/opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py'.
> [Tue Mar 27 12:16:30.351474 2018] [wsgi:error] [pid 11487] [client
> 192.168.1.100:21802] Truncated or oversized response headers received from
> daemon process 'project_name':
> /opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py
> [Tue Mar 27 12:16:30.961282 2018] [wsgi:info] [pid 11570] mod_wsgi
> (pid=11570): Attach interpreter ''.
> [Tue Mar 27 12:16:30.961504 2018] [wsgi:info] [pid 11570] mod_wsgi
> (pid=11570): Adding '/opt/rh/httpd24/root/var/www/html/sitea' to path.
> [Tue Mar 27 12:16:30.963586 2018] [wsgi:info] [pid 11570] mod_wsgi
> (pid=11570): Imported 'mod_wsgi'.
> [Tue Mar 27 12:16:47.982091 2018] [wsgi:info] [pid 11570] [remote
> 192.168.1.100:21804] mod_wsgi (pid=11570, process='project_name',
> application=''): Loading Python script file
> '/opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py'.
> [Tue Mar 27 12:16:48.472366 2018] [wsgi:error] [pid 11492] [client
> 192.168.1.100:21804] Truncated or oversized response headers received from
> daemon process 'project_name':
> /opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py
> [Tue Mar 27 12:16:48.979670 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Attach interpreter ''.
> [Tue Mar 27 12:16:48.979908 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Adding '/opt/rh/httpd24/root/var/www/html/sitea' to path.
> [Tue Mar 27 12:16:48.982001 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Imported 'mod_wsgi'.
> [Tue Mar 27 12:17:14.221963 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Shutdown requested 'project_name'.
> [Tue Mar 27 12:17:14.222210 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Stopping process 'project_name'.
> [Tue Mar 27 12:17:14.222230 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Destroying interpreters.
> [Tue Mar 27 12:17:14.222242 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Cleanup interpreter ''.
> [Tue Mar 27 12:17:14.228896 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Terminating Python.
> [Tue Mar 27 12:17:14.232279 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Python has shutdown.
> [Tue Mar 27 12:17:14.232304 2018] [wsgi:info] [pid 11607] mod_wsgi
> (pid=11607): Exiting process 'project_name'.
>
> The site is connecting via SSL and verifying client certificate before
> forwarding to the Django app. The SSL portion works as expected. Once it
> validates the client certificate, it receives a 500 error. I've removed the
> ssl information for privacy reasons.
>
> As far as the code content under the directory, I understand it isn't a best
> practice, but it isn't an aspect I can change at the moment. System is
> operating in a corporate offline network.
>
> On Tuesday, March 27, 2018 at 6:00:00 AM UTC-4, Graham Dumpleton wrote:
>
>
>> On 27 Mar 2018, at 8:35 pm, Eric Sears <ese...@ <>gmail.com
>> <http://gmail.com/>> wrote:
>>
>> Are you modifying sys.path in the WSGI script file so Python will know where
>> your project code is?
>>
>> You have to excuse me as I didn't build the Django app. The main wsgi.py
>> calls the site.settings file. In the settings.py the closest thing I could
>> find is:
>>
>> BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
>>
>>
>> Do you mean using:
>>
>> WSGIApplicationGroup %{GLOBAL}
>>
>> It is preferable you use that regardless.
>> The information in the virtual host settings is:
>
> Which is different to what you showed before.
>
>>
>> WSGIDaemonProcess projectname
>> python-path=/opt/rh/httpd24/root/var/www/html/sitea/project_name
>
> The python-path setting here is an alternative for setting sys.path in the
> wsgi.py file.
>
> Note that it looks like you have placed all your code under DocumentRoot
> directory. That is:
>
> /opt/rh/httpd24/root/var/www/html
>
> That is a bad idea. If someone commented out WSGIScriptAlias, your code can
> be downloaded, including any passwords of the secret key in the Django
> settings file.
>
> You should not put your Django project code under the directory specified by
> DocumentRoot.
>
>> WSGIProcessGroup project_name
>> WSGIScriptAlias /
>> /opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py
>> application-group=%{GLOBAL}
>
> So using application-group instead of WSGIApplicationGroup.
>
>> Can you ensure you have set:
>>
>> LogLevel info
>>
>> in Apache configuration.
>>
>> LogLevel is currently set to debug as I wanted to get as much info as
>> possible.
>
> Using info is sufficient. Plus the logs below indicate you might not be using
> debug when that request was made as I would have expected there to be more
> information than that, including some debug message. Anyway, doesn't matter,
> whether debug messages show may depend on other Apache modules being used.
> Extra messages from mod_wsgi may only get displayed if also have
> WSGIVerboseDebugging directive set to On, but don't need to see them.
>
>> This will cause mod_wsgi to log more about process restarts, script loading
>> etc. I need to see the more detailed logging from the log file around time
>> the request was made. If you are using VirtualHost specific log, also need
>> to see any errors from main
>> Apache error log at the same time.
>>
>> This is a disconnected system and I can't export the logs directly.
>> Hopefully the info below is what you need:
>>
>> [wsgi:info] [pid 27937:tid #] mod_wsgi (pid=27937): Attach interpreter ''.
>> [wsgi.info <http://wsgi.info/>] [pid 27937:tid #] mod_wsgi (pid=27937):
>> Adding '/opt/rh/httpd24/root/var/www/html/sitea' to path
>> [wsgi:debug] - Spawns 15 threads in daemon process ' project_name'
>> [wsgi:info] [pid 27937:tid #] [remote 192.168.1.100] mod_wsgi (pid 27937,
>> process='project_name', application=''): Loading Python script file
>> '/opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py'
>> [wsgi:error] [pid 27981] [client 192.168.1.100] Truncated or oversized
>> response headers received from damon process 'project_name':
>> /opt/rh/httpd24/root/var/www/html/sitea/project_name/wsgi.py, referrer
>> https://site.com/ <https://site.com/>
>
> Are you sure there is nothing after this?
>
> Can you do more than the one request and show me error log messages from
> across the two requests.
>
> Are you sure there are no messages showing a message like 'Segmentation
> Fault'? Does the process ID shown in the 'Loading Python script file' message
> in logs still exist after you make the request?
>
> Does your application return a WWW-Authenticate header with a really long
> authentication challenge string?
>
> Does your application return any other response headers with really long
> values?
>
> If yes to either of last two questions, try adding:
>
> header-buffer-size=16384
>
> option to WSGIDaemonProcess directive.
>
> Graham
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To post to this group, send email to [email protected]
> <mailto:[email protected]>.
> Visit this group at https://groups.google.com/group/modwsgi
> <https://groups.google.com/group/modwsgi>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
