One quick question while I think about what changed between 4.1.3 and 4.2.4.
When you install a newer version of mod_wsgi, does it get compiled from source code against the specific Apache installation you are using, or are you using a binary package? If using a binary package for mod_wsgi and it was compiled for a newer version of Apache than you are actually using, that can be a source of problems. Is the Apache version you are using (which is actually very old), the latest Apache version available for your distribution? In general I would only recommend using at least Apache 2.2.18ish as minimum as there were some issues in Apache which were fixed around Apache 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the reports seemed to vanish for newer versions. Exactly what the problem was I am not sure, but believe it was related to SSL changes in Apache. Graham On 13/07/2014, at 12:23 PM, Allen Lee <[email protected]> wrote: > Hi Graham, > > Thank you for the fast response! I've been checking to see what happened > prior and it looks the relevant packages that changed between the working -> > non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors > started after logrotate reloaded apache over the weekend. Responses to > questions included inline: > > > On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote: > What did you do just prior to the weekend that changed anything? > > This must have been caused by a package update earlier in the weekj that > manifested itself after httpd restarted at 4 AM this morning from a weekly > logrotate. Possibly relevant entries from yum.log: > Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64 > Jun 04 14:16:43 Installed: > python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch > Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch > Jun 04 14:16:52 Erased: python27-distribute > Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64 > Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64 > Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64 > Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch > Jun 09 15:05:59 Erased: python27-virtualenv > Jun 09 15:06:10 Erased: python-virtualenv > Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64 > Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64 > Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64 > Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64 > Jun 20 14:53:49 Updated: > python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch > Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch > Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64 > Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch > Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 > Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch > Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 > Jul 13 08:53:16 Installed: python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64 > Jul 13 09:06:15 Erased: subversion-python > > > Did you upgrade the Python installation you are dependent on, but not > recreate the Python virtual environment against the updated Python > installation? > > I just checked this and noticed that my virtualenv Python was indeed > different from the updated Python installation. I recreated the Python > virtualenv using the updated system Python 2.7 (now there is no diff between > virtualenv Python and system Python) and restarted the server but it still > continually segfaults like it used to. > > > Did you apply any update to the Apache version being used? > > Nope. > > > Did you change the Apache configuration to load any additional Apache modules? > > Nope. > > > Did you change the Apache configuration for any modules you were already > using? > > Nope. > > > How many Python installations exist on the system and where are they > installed? > > There are actually 3 installs of Python on this antiquated VM, the stock RHEL > 5 python 2.4, python2.6 (which I can remove though it would take byobu with > it which I rather like), and python2.7 from the IUS repositories. Python 2.4 > and 2.6 are RHEL stock installs, 2.7 is from the IUS repo. > > > What Python installation is your Python virtual environment based off? > > Python 2.7 > > > What Python installation was mod_wsgi compiled against? > > Python2.7, using the python27-mod_wsgi from the IUS repo as well. > > > Is the mod_wsgi.so finding the correct Python shared library for the Python > installation you expect it to use? > > http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library > > Yes, I did check this and it appears to be correct: > # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so > linux-vdso.so.1 => (0x00007fff7e577000) > libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 > (0x00002b99ffd5d000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000) > libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000) > libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000) > libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000) > libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000) > /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000) > > > Are you using mod_wsgi daemon mode? > > Yes. I tried switching to embedded mode but it exhibits the same behavior. > Here's the wsgi related config from my Apache vhost: > > WSGISocketPrefix run/wsgi > WSGIPythonHome /opt/virtualenvs/vcweb > <VirtualHost *:80> > ... > WSGIDaemonProcess vcweb user=apache group=commons threads=25 > python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages > WSGIProcessGroup vcweb > WSGIScriptAlias / /opt/vcweb/wsgi.py > ... > </VirtualHost> > > > Is Apache segfaulting on startup in all child processes, or only for mod_wsgi > daemon mode processes? > > It appears that it's only for mod_wsgi daemon mode processes, php and Java > sites also served by the same webserver are still up. > > > Is the segfault truly on startup, or only on the first request against a > process? > > Yes, as soon as the server starts up my base httpd error.log starts spamming > child pid segfaults (exit 11) and the vhost error.log spams messages along > these lines: > > [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter ''. > [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/vcweb' > to path. > [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding > '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. > > > Sorry for all the questions, but there isn't a lot to go on at this point. > > The point of these questions is to prompt you to validate certain things > about your installation and ensure they are correct, but responses to them > will also help me to understand your setup and so be able to work out what > the issue may be. > > Thank you for taking a look, much appreciated! I've been planning to upgrade > these VMs to RHEL 7 this summer anyways so there's no need to plunge a lot of > resources into this if you don't see anything immediately untoward. > > > Graham > > On 13/07/2014, at 9:47 AM, Allen Lee <[email protected]> wrote: > >> (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but wasn't >> sure where the appropriate place for this is) >> >> I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS >> repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running >> within a virtualenv. I haven't had any problems until this weekend when >> Apache has started to segfault constantly when trying to load the python >> interpreter - messages like (child pid 12574 exit signal Segmentation fault >> (11)). >> >> I've read through >> https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes >> and can confirm that: >> >> 1/ mod_python is not installed, and >> >> 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running >> 2.0.1 but I didn't encounter a segfault when running: >> >> (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python >> Python 2.7.7 (default, Jun 4 2014, 17:09:35) >> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import pyexpat >> >>> pyexpat.version_info >> (1, 95, 8) >> >>> >> >> I dumped the core and extracted the following from gdb but am at a loss at >> this point, any additional pointers to try to identify the source of the >> problem would be much appreciated! >> >> Results of gdb /usr/sbin/httpd /path/to/core.dump: >> <snipped> >> warning: no loadable sections found in added symbol-file system-supplied DSO >> at 0x7fff1dffd000 >> Core was generated by `/usr/sbin/httpd'. >> Program terminated with signal 11, Segmentation fault. >> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >> (gdb) where >> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >> #1 0x00002aed4a24e43b in PyString_FromString () from >> /usr/lib64/libpython2.7.so.1.0 >> #2 0x00002aed49f8124b in ?? () >> #3 0x00002aed00005168 in ?? () >> #4 0x00002aed49f97190 in ?? () >> #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0 >> #6 0x00000000ffffffff in ?? () >> #7 0x00002aed49f97190 in ?? () >> #8 0x00002aed3d826cf0 in ?? () >> #9 0x00002aed4a1a0b98 in ?? () >> #10 0x0000000000000001 in ?? () >> #11 0x00002aed3d826cf0 in ?? () >> #12 0x0000000000000000 in ?? () >> (gdb) >> >> # httpd -V >> Server version: Apache/2.2.3 >> Server built: Mar 26 2014 08:47:55 >> Server's Module Magic Number: 20051115:3 >> Server loaded: APR 1.5.0, APR-Util 1.5.3 >> Compiled using: APR 1.2.7, APR-Util 1.2.7 >> Architecture: 64-bit >> Server MPM: Prefork >> threaded: no >> forked: yes (variable process count) >> Server compiled with.... >> -D APACHE_MPM_DIR="server/mpm/prefork" >> -D APR_HAS_SENDFILE >> -D APR_HAS_MMAP >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >> -D APR_USE_SYSVSEM_SERIALIZE >> -D APR_USE_PTHREAD_SERIALIZE >> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >> -D APR_HAS_OTHER_CHILD >> -D AP_HAVE_RELIABLE_PIPED_LOGS >> -D DYNAMIC_MODULE_LIMIT=128 >> -D HTTPD_ROOT="/etc/httpd" >> -D SUEXEC_BIN="/usr/sbin/suexec" >> -D DEFAULT_PIDLOG="run/httpd.pid" >> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >> -D DEFAULT_LOCKFILE="logs/accept.lock" >> -D DEFAULT_ERRORLOG="logs/error_log" >> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >> -D SERVER_CONFIG_FILE="conf/httpd.conf" >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
