Hi Graham, Thank you for the fast response! I've been checking to see what happened prior and it looks the relevant packages that changed between the working -> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors started after logrotate reloaded apache over the weekend. Responses to questions included inline:
On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote: > > What did you do just prior to the weekend that changed anything? > This must have been caused by a package update earlier in the weekj that manifested itself after httpd restarted at 4 AM this morning from a weekly logrotate. Possibly relevant entries from yum.log: Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64 Jun 04 14:16:43 Installed: python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch Jun 04 14:16:52 Erased: python27-distribute Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64 Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64 Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64 Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch Jun 09 15:05:59 Erased: python27-virtualenv Jun 09 15:06:10 Erased: python-virtualenv Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64 Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64 Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64 Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64 Jun 20 14:53:49 Updated: python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64 Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 Jul 13 08:53:16 Installed: python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64 Jul 13 09:06:15 Erased: subversion-python > > Did you upgrade the Python installation you are dependent on, but not > recreate the Python virtual environment against the updated Python > installation? > I just checked this and noticed that my virtualenv Python was indeed different from the updated Python installation. I recreated the Python virtualenv using the updated system Python 2.7 (now there is no diff between virtualenv Python and system Python) and restarted the server but it still continually segfaults like it used to. > > Did you apply any update to the Apache version being used? > Nope. > > Did you change the Apache configuration to load any additional Apache > modules? > Nope. > Did you change the Apache configuration for any modules you were already > using? > Nope. > How many Python installations exist on the system and where are they > installed? > There are actually 3 installs of Python on this antiquated VM, the stock RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu with it which I rather like), and python2.7 from the IUS repositories. Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo. > > What Python installation is your Python virtual environment based off? > Python 2.7 > > What Python installation was mod_wsgi compiled against? > Python2.7, using the python27-mod_wsgi from the IUS repo as well. > > Is the mod_wsgi.so finding the correct Python shared library for the > Python installation you expect it to use? > > > http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library > Yes, I did check this and it appears to be correct: # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so linux-vdso.so.1 => (0x00007fff7e577000) libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00002b99ffd5d000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000) libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000) libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000) /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000) > > Are you using mod_wsgi daemon mode? > Yes. I tried switching to embedded mode but it exhibits the same behavior. Here's the wsgi related config from my Apache vhost: WSGISocketPrefix run/wsgi WSGIPythonHome /opt/virtualenvs/vcweb <VirtualHost *:80> ... WSGIDaemonProcess vcweb user=apache group=commons threads=25 python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages WSGIProcessGroup vcweb WSGIScriptAlias / /opt/vcweb/wsgi.py ... </VirtualHost> > Is Apache segfaulting on startup in all child processes, or only for > mod_wsgi daemon mode processes? > It appears that it's only for mod_wsgi daemon mode processes, php and Java sites also served by the same webserver are still up. > Is the segfault truly on startup, or only on the first request against a > process? > Yes, as soon as the server starts up my base httpd error.log starts spamming child pid segfaults (exit 11) and the vhost error.log spams messages along these lines: [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter ''. [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/vcweb' to path. [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. > Sorry for all the questions, but there isn't a lot to go on at this point. > > The point of these questions is to prompt you to validate certain things > about your installation and ensure they are correct, but responses to them > will also help me to understand your setup and so be able to work out what > the issue may be. > Thank you for taking a look, much appreciated! I've been planning to upgrade these VMs to RHEL 7 this summer anyways so there's no need to plunge a lot of resources into this if you don't see anything immediately untoward. > > Graham > > On 13/07/2014, at 9:47 AM, Allen Lee <[email protected] <javascript:>> > wrote: > > (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but > wasn't sure where the appropriate place for this is) > > I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS > repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running > within a virtualenv. I haven't had any problems until this weekend when > Apache has started to segfault constantly when trying to load the python > interpreter - messages like (child pid 12574 exit signal Segmentation fault > (11)). > > I've read through > https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes > > and can confirm that: > > 1/ mod_python is not installed, and > > 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running > 2.0.1 but I didn't encounter a segfault when running: > > (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python > Python 2.7.7 (default, Jun 4 2014, 17:09:35) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import pyexpat > >>> pyexpat.version_info > (1, 95, 8) > >>> > > I dumped the core and extracted the following from gdb but am at a loss at > this point, any additional pointers to try to identify the source of the > problem would be much appreciated! > > Results of gdb /usr/sbin/httpd /path/to/core.dump: > <snipped> > warning: no loadable sections found in added symbol-file system-supplied > DSO at 0x7fff1dffd000 > Core was generated by `/usr/sbin/httpd'. > Program terminated with signal 11, Segmentation fault. > #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 > (gdb) where > #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 > #1 0x00002aed4a24e43b in PyString_FromString () from > /usr/lib64/libpython2.7.so.1.0 > #2 0x00002aed49f8124b in ?? () > #3 0x00002aed00005168 in ?? () > #4 0x00002aed49f97190 in ?? () > #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0 > #6 0x00000000ffffffff in ?? () > #7 0x00002aed49f97190 in ?? () > #8 0x00002aed3d826cf0 in ?? () > #9 0x00002aed4a1a0b98 in ?? () > #10 0x0000000000000001 in ?? () > #11 0x00002aed3d826cf0 in ?? () > #12 0x0000000000000000 in ?? () > (gdb) > > # httpd -V > Server version: Apache/2.2.3 > Server built: Mar 26 2014 08:47:55 > Server's Module Magic Number: 20051115:3 > Server loaded: APR 1.5.0, APR-Util 1.5.3 > Compiled using: APR 1.2.7, APR-Util 1.2.7 > Architecture: 64-bit > Server MPM: Prefork > threaded: no > forked: yes (variable process count) > Server compiled with.... > -D APACHE_MPM_DIR="server/mpm/prefork" > -D APR_HAS_SENDFILE > -D APR_HAS_MMAP > -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) > -D APR_USE_SYSVSEM_SERIALIZE > -D APR_USE_PTHREAD_SERIALIZE > -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT > -D APR_HAS_OTHER_CHILD > -D AP_HAVE_RELIABLE_PIPED_LOGS > -D DYNAMIC_MODULE_LIMIT=128 > -D HTTPD_ROOT="/etc/httpd" > -D SUEXEC_BIN="/usr/sbin/suexec" > -D DEFAULT_PIDLOG="run/httpd.pid" > -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" > -D DEFAULT_LOCKFILE="logs/accept.lock" > -D DEFAULT_ERRORLOG="logs/error_log" > -D AP_TYPES_CONFIG_FILE="conf/mime.types" > -D SERVER_CONFIG_FILE="conf/httpd.conf" > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
