Can you ensure that LogLevel directive in Apache is set to 'debug'. Also add the directive:
WSGIVerboseDebugging On at global scope. The give me a snippet of the log messages from Apache/mod_wsgi that lead up to the segfault. I can possibly narrow it down from that. Graham On 13/07/2014, at 12:47 PM, Graham Dumpleton <[email protected]> wrote: > One quick question while I think about what changed between 4.1.3 and 4.2.4. > > When you install a newer version of mod_wsgi, does it get compiled from > source code against the specific Apache installation you are using, or are > you using a binary package? > > If using a binary package for mod_wsgi and it was compiled for a newer > version of Apache than you are actually using, that can be a source of > problems. > > Is the Apache version you are using (which is actually very old), the latest > Apache version available for your distribution? > > In general I would only recommend using at least Apache 2.2.18ish as minimum > as there were some issues in Apache which were fixed around Apache > 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the > reports seemed to vanish for newer versions. Exactly what the problem was I > am not sure, but believe it was related to SSL changes in Apache. > > Graham > > On 13/07/2014, at 12:23 PM, Allen Lee <[email protected]> wrote: > >> Hi Graham, >> >> Thank you for the fast response! I've been checking to see what happened >> prior and it looks the relevant packages that changed between the working -> >> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors >> started after logrotate reloaded apache over the weekend. Responses to >> questions included inline: >> >> >> On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote: >> What did you do just prior to the weekend that changed anything? >> >> This must have been caused by a package update earlier in the weekj that >> manifested itself after httpd restarted at 4 AM this morning from a weekly >> logrotate. Possibly relevant entries from yum.log: >> Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64 >> Jun 04 14:16:43 Installed: >> python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch >> Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch >> Jun 04 14:16:52 Erased: python27-distribute >> Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64 >> Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64 >> Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64 >> Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch >> Jun 09 15:05:59 Erased: python27-virtualenv >> Jun 09 15:06:10 Erased: python-virtualenv >> Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64 >> Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64 >> Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64 >> Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64 >> Jun 20 14:53:49 Updated: >> python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch >> Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch >> Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64 >> Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch >> Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 >> Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch >> Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 >> Jul 13 08:53:16 Installed: python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64 >> Jul 13 09:06:15 Erased: subversion-python >> >> >> Did you upgrade the Python installation you are dependent on, but not >> recreate the Python virtual environment against the updated Python >> installation? >> >> I just checked this and noticed that my virtualenv Python was indeed >> different from the updated Python installation. I recreated the Python >> virtualenv using the updated system Python 2.7 (now there is no diff between >> virtualenv Python and system Python) and restarted the server but it still >> continually segfaults like it used to. >> >> >> Did you apply any update to the Apache version being used? >> >> Nope. >> >> >> Did you change the Apache configuration to load any additional Apache >> modules? >> >> Nope. >> >> >> Did you change the Apache configuration for any modules you were already >> using? >> >> Nope. >> >> >> How many Python installations exist on the system and where are they >> installed? >> >> There are actually 3 installs of Python on this antiquated VM, the stock >> RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu >> with it which I rather like), and python2.7 from the IUS repositories. >> Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo. >> >> >> What Python installation is your Python virtual environment based off? >> >> Python 2.7 >> >> >> What Python installation was mod_wsgi compiled against? >> >> Python2.7, using the python27-mod_wsgi from the IUS repo as well. >> >> >> Is the mod_wsgi.so finding the correct Python shared library for the Python >> installation you expect it to use? >> >> http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library >> >> Yes, I did check this and it appears to be correct: >> # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so >> linux-vdso.so.1 => (0x00007fff7e577000) >> libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 >> (0x00002b99ffd5d000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000) >> libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000) >> libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000) >> libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000) >> /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000) >> >> >> Are you using mod_wsgi daemon mode? >> >> Yes. I tried switching to embedded mode but it exhibits the same behavior. >> Here's the wsgi related config from my Apache vhost: >> >> WSGISocketPrefix run/wsgi >> WSGIPythonHome /opt/virtualenvs/vcweb >> <VirtualHost *:80> >> ... >> WSGIDaemonProcess vcweb user=apache group=commons threads=25 >> python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages >> WSGIProcessGroup vcweb >> WSGIScriptAlias / /opt/vcweb/wsgi.py >> ... >> </VirtualHost> >> >> >> Is Apache segfaulting on startup in all child processes, or only for >> mod_wsgi daemon mode processes? >> >> It appears that it's only for mod_wsgi daemon mode processes, php and Java >> sites also served by the same webserver are still up. >> >> >> Is the segfault truly on startup, or only on the first request against a >> process? >> >> Yes, as soon as the server starts up my base httpd error.log starts spamming >> child pid segfaults (exit 11) and the vhost error.log spams messages along >> these lines: >> >> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter >> ''. >> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/vcweb' >> to path. >> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding >> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. >> >> >> Sorry for all the questions, but there isn't a lot to go on at this point. >> >> The point of these questions is to prompt you to validate certain things >> about your installation and ensure they are correct, but responses to them >> will also help me to understand your setup and so be able to work out what >> the issue may be. >> >> Thank you for taking a look, much appreciated! I've been planning to upgrade >> these VMs to RHEL 7 this summer anyways so there's no need to plunge a lot >> of resources into this if you don't see anything immediately untoward. >> >> >> Graham >> >> On 13/07/2014, at 9:47 AM, Allen Lee <[email protected]> wrote: >> >>> (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but wasn't >>> sure where the appropriate place for this is) >>> >>> I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS >>> repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running >>> within a virtualenv. I haven't had any problems until this weekend when >>> Apache has started to segfault constantly when trying to load the python >>> interpreter - messages like (child pid 12574 exit signal Segmentation fault >>> (11)). >>> >>> I've read through >>> https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes >>> and can confirm that: >>> >>> 1/ mod_python is not installed, and >>> >>> 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running >>> 2.0.1 but I didn't encounter a segfault when running: >>> >>> (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python >>> Python 2.7.7 (default, Jun 4 2014, 17:09:35) >>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> import pyexpat >>> >>> pyexpat.version_info >>> (1, 95, 8) >>> >>> >>> >>> I dumped the core and extracted the following from gdb but am at a loss at >>> this point, any additional pointers to try to identify the source of the >>> problem would be much appreciated! >>> >>> Results of gdb /usr/sbin/httpd /path/to/core.dump: >>> <snipped> >>> warning: no loadable sections found in added symbol-file system-supplied >>> DSO at 0x7fff1dffd000 >>> Core was generated by `/usr/sbin/httpd'. >>> Program terminated with signal 11, Segmentation fault. >>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >>> (gdb) where >>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >>> #1 0x00002aed4a24e43b in PyString_FromString () from >>> /usr/lib64/libpython2.7.so.1.0 >>> #2 0x00002aed49f8124b in ?? () >>> #3 0x00002aed00005168 in ?? () >>> #4 0x00002aed49f97190 in ?? () >>> #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0 >>> #6 0x00000000ffffffff in ?? () >>> #7 0x00002aed49f97190 in ?? () >>> #8 0x00002aed3d826cf0 in ?? () >>> #9 0x00002aed4a1a0b98 in ?? () >>> #10 0x0000000000000001 in ?? () >>> #11 0x00002aed3d826cf0 in ?? () >>> #12 0x0000000000000000 in ?? () >>> (gdb) >>> >>> # httpd -V >>> Server version: Apache/2.2.3 >>> Server built: Mar 26 2014 08:47:55 >>> Server's Module Magic Number: 20051115:3 >>> Server loaded: APR 1.5.0, APR-Util 1.5.3 >>> Compiled using: APR 1.2.7, APR-Util 1.2.7 >>> Architecture: 64-bit >>> Server MPM: Prefork >>> threaded: no >>> forked: yes (variable process count) >>> Server compiled with.... >>> -D APACHE_MPM_DIR="server/mpm/prefork" >>> -D APR_HAS_SENDFILE >>> -D APR_HAS_MMAP >>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>> -D APR_USE_SYSVSEM_SERIALIZE >>> -D APR_USE_PTHREAD_SERIALIZE >>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>> -D APR_HAS_OTHER_CHILD >>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>> -D DYNAMIC_MODULE_LIMIT=128 >>> -D HTTPD_ROOT="/etc/httpd" >>> -D SUEXEC_BIN="/usr/sbin/suexec" >>> -D DEFAULT_PIDLOG="run/httpd.pid" >>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>> -D DEFAULT_LOCKFILE="logs/accept.lock" >>> -D DEFAULT_ERRORLOG="logs/error_log" >>> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
