Hi Graham, I added the two files - it does emit "Imported 'mod_wsgi'." but *not* "Imported 'apache'."
My apache config does indeed have ServerSignature On set properly. One thing strange I noticed related to ap_get_server_description - after I removed mod_perl v2.0.4-6 in an effort to remove unused/unneeded modules, httpd wouldn't even start if python27-mod_wsgi.so was loaded, complaining "Cannot load /etc/httpd/modules/python27-mod_wsgi.so into server: undefined symbol: ap_get_server_description". Removing the mod_wsgi load without mod_perl installed allowed httpd to start up normally. I'm not sure if this is related or not though... On Sun, Jul 13, 2014 at 8:39 PM, Graham Dumpleton <[email protected]> wrote: > Can you add to the directory: > > /opt/vcweb > > two empty files called: > > mod_wsgi.py > apache.py > > and restart and look then to see if the debug logging outputs: > > mod_wsgi (pid=%d): Imported 'mod_wsgi'. > mod_wsgi (pid=%d): Imported 'apache'. > > If they are output, then it I know it is getting past that point. > > I suspect it may be dying in the following code which was changed in the > newer version. > > str = ap_get_server_description(); > #if PY_MAJOR_VERSION >= 3 > object = PyUnicode_DecodeLatin1(str, strlen(str), NULL); > #else > object = PyString_FromString(str); > #endif > PyModule_AddObject(module, "description", object); > > str = MPM_NAME; > #if PY_MAJOR_VERSION >= 3 > object = PyUnicode_DecodeLatin1(str, strlen(str), NULL); > #else > object = PyString_FromString(str); > #endif > PyModule_AddObject(module, "mpm_name", object); > > str = ap_get_server_built(); > #if PY_MAJOR_VERSION >= 3 > object = PyUnicode_DecodeLatin1(str, strlen(str), NULL); > #else > object = PyString_FromString(str); > #endif > PyModule_AddObject(module, "build_date", object); > > Either the old Apache is returning NULL for some of the Apache calls, or > Redhat in modifying the server to suppress the server description, is > causing a NULL to be returned. > > What you might do is check if your Apache configuration has: > > ServerSignature Off > > and set it back to: > > ServerSignature On > > and try again. > > Even with that as Off it still should return an empty string and not a NULL > though. But then, for a standard Apache description, that flag should affect > the response from ap_get_server_description() anyway. > > Graham > > On 13/07/2014, at 2:17 PM, Allen Lee <[email protected]> wrote: > > I'm pretty sure it is a binary package that gets installed, though I'm not > sure what version of httpd it is getting compiled against, I would presume > it is the stock RHEL 5 httpd though. It is the latest version of httpd for > RHEL 5. I've provisioned a new RHEL 7 instance and mod_wsgi is working fine > there, so I may just chalk this up as the final straw for finally upgrading > our VMs. > > The LogLevel is set to debug and I added the WSGIVerboseDebugging but didn't > get any different output in the root error.log (constant segfaults from > child pid) and vhost error.log.. > > [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Attach interpreter > ''. > > [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding '/opt/vcweb' > to path. > > [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding > '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. > > [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Attach interpreter > ''. > > [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding '/opt/vcweb' > to path. > > [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding > '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. > > On Sunday, July 13, 2014 1:13:04 PM UTC-7, Graham Dumpleton wrote: >> >> Can you ensure that LogLevel directive in Apache is set to 'debug'. >> >> Also add the directive: >> >> WSGIVerboseDebugging On >> >> at global scope. >> >> The give me a snippet of the log messages from Apache/mod_wsgi that lead >> up to the segfault. >> >> I can possibly narrow it down from that. >> >> Graham >> >> On 13/07/2014, at 12:47 PM, Graham Dumpleton <[email protected]> >> wrote: >> >> One quick question while I think about what changed between 4.1.3 and >> 4.2.4. >> >> When you install a newer version of mod_wsgi, does it get compiled from >> source code against the specific Apache installation you are using, or are >> you using a binary package? >> >> If using a binary package for mod_wsgi and it was compiled for a newer >> version of Apache than you are actually using, that can be a source of >> problems. >> >> Is the Apache version you are using (which is actually very old), the >> latest Apache version available for your distribution? >> >> In general I would only recommend using at least Apache 2.2.18ish as >> minimum as there were some issues in Apache which were fixed around Apache >> 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the >> reports seemed to vanish for newer versions. Exactly what the problem was I >> am not sure, but believe it was related to SSL changes in Apache. >> >> Graham >> >> On 13/07/2014, at 12:23 PM, Allen Lee <[email protected]> wrote: >> >> Hi Graham, >> >> Thank you for the fast response! I've been checking to see what happened >> prior and it looks the relevant packages that changed between the working -> >> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors >> started after logrotate reloaded apache over the weekend. Responses to >> questions included inline: >> >> >> On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote: >>> >>> What did you do just prior to the weekend that changed anything? >> >> >> This must have been caused by a package update earlier in the weekj that >> manifested itself after httpd restarted at 4 AM this morning from a weekly >> logrotate. Possibly relevant entries from yum.log: >> Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64 >> Jun 04 14:16:43 Installed: >> python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch >> Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch >> Jun 04 14:16:52 Erased: python27-distribute >> Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64 >> Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64 >> Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64 >> Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch >> Jun 09 15:05:59 Erased: python27-virtualenv >> Jun 09 15:06:10 Erased: python-virtualenv >> Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64 >> Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64 >> Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64 >> Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64 >> Jun 20 14:53:49 Updated: >> python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch >> Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch >> Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64 >> Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch >> Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 >> Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch >> Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 >> Jul 13 08:53:16 Installed: >> python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64 >> Jul 13 09:06:15 Erased: subversion-python >> >>> >>> >>> Did you upgrade the Python installation you are dependent on, but not >>> recreate the Python virtual environment against the updated Python >>> installation? >> >> >> I just checked this and noticed that my virtualenv Python was indeed >> different from the updated Python installation. I recreated the Python >> virtualenv using the updated system Python 2.7 (now there is no diff between >> virtualenv Python and system Python) and restarted the server but it still >> continually segfaults like it used to. >> >>> >>> >>> Did you apply any update to the Apache version being used? >> >> >> Nope. >> >>> >>> >>> Did you change the Apache configuration to load any additional Apache >>> modules? >> >> >> Nope. >> >>> >>> Did you change the Apache configuration for any modules you were already >>> using? >> >> >> Nope. >> >>> >>> How many Python installations exist on the system and where are they >>> installed? >> >> >> There are actually 3 installs of Python on this antiquated VM, the stock >> RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu >> with it which I rather like), and python2.7 from the IUS repositories. >> Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo. >> >>> >>> >>> What Python installation is your Python virtual environment based off? >> >> >> Python 2.7 >> >>> >>> >>> What Python installation was mod_wsgi compiled against? >> >> >> Python2.7, using the python27-mod_wsgi from the IUS repo as well. >> >>> >>> >>> Is the mod_wsgi.so finding the correct Python shared library for the >>> Python installation you expect it to use? >>> >>> >>> >>> http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library >> >> >> Yes, I did check this and it appears to be correct: >> # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so >> linux-vdso.so.1 => (0x00007fff7e577000) >> libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 >> (0x00002b99ffd5d000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000) >> libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000) >> libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000) >> libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000) >> /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000) >> >>> >>> >>> Are you using mod_wsgi daemon mode? >> >> >> Yes. I tried switching to embedded mode but it exhibits the same behavior. >> Here's the wsgi related config from my Apache vhost: >> >> WSGISocketPrefix run/wsgi >> WSGIPythonHome /opt/virtualenvs/vcweb >> <VirtualHost *:80> >> ... >> WSGIDaemonProcess vcweb user=apache group=commons threads=25 >> python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages >> WSGIProcessGroup vcweb >> WSGIScriptAlias / /opt/vcweb/wsgi.py >> ... >> </VirtualHost> >> >>> >>> Is Apache segfaulting on startup in all child processes, or only for >>> mod_wsgi daemon mode processes? >> >> >> It appears that it's only for mod_wsgi daemon mode processes, php and Java >> sites also served by the same webserver are still up. >> >>> >>> Is the segfault truly on startup, or only on the first request against a >>> process? >> >> >> Yes, as soon as the server starts up my base httpd error.log starts >> spamming child pid segfaults (exit 11) and the vhost error.log spams >> messages along these lines: >> >> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter >> ''. >> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding >> '/opt/vcweb' to path. >> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding >> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. >> >>> >>> Sorry for all the questions, but there isn't a lot to go on at this >>> point. >>> >>> >>> The point of these questions is to prompt you to validate certain things >>> about your installation and ensure they are correct, but responses to them >>> will also help me to understand your setup and so be able to work out what >>> the issue may be. >> >> >> Thank you for taking a look, much appreciated! I've been planning to >> upgrade these VMs to RHEL 7 this summer anyways so there's no need to plunge >> a lot of resources into this if you don't see anything immediately untoward. >> >>> >>> >>> Graham >>> >>> On 13/07/2014, at 9:47 AM, Allen Lee <[email protected]> wrote: >>> >>> (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but >>> wasn't sure where the appropriate place for this is) >>> >>> I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS >>> repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running >>> within a virtualenv. I haven't had any problems until this weekend when >>> Apache has started to segfault constantly when trying to load the python >>> interpreter - messages like (child pid 12574 exit signal Segmentation fault >>> (11)). >>> >>> I've read through >>> https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes >>> and can confirm that: >>> >>> 1/ mod_python is not installed, and >>> >>> 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running >>> 2.0.1 but I didn't encounter a segfault when running: >>> >>> (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python >>> Python 2.7.7 (default, Jun 4 2014, 17:09:35) >>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> import pyexpat >>> >>> pyexpat.version_info >>> (1, 95, 8) >>> >>> >>> >>> I dumped the core and extracted the following from gdb but am at a loss >>> at this point, any additional pointers to try to identify the source of the >>> problem would be much appreciated! >>> >>> Results of gdb /usr/sbin/httpd /path/to/core.dump: >>> <snipped> >>> warning: no loadable sections found in added symbol-file system-supplied >>> DSO at 0x7fff1dffd000 >>> Core was generated by `/usr/sbin/httpd'. >>> Program terminated with signal 11, Segmentation fault. >>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >>> (gdb) where >>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >>> #1 0x00002aed4a24e43b in PyString_FromString () from >>> /usr/lib64/libpython2.7.so.1.0 >>> #2 0x00002aed49f8124b in ?? () >>> #3 0x00002aed00005168 in ?? () >>> #4 0x00002aed49f97190 in ?? () >>> #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0 >>> #6 0x00000000ffffffff in ?? () >>> #7 0x00002aed49f97190 in ?? () >>> #8 0x00002aed3d826cf0 in ?? () >>> #9 0x00002aed4a1a0b98 in ?? () >>> #10 0x0000000000000001 in ?? () >>> #11 0x00002aed3d826cf0 in ?? () >>> #12 0x0000000000000000 in ?? () >>> (gdb) >>> >>> # httpd -V >>> Server version: Apache/2.2.3 >>> Server built: Mar 26 2014 08:47:55 >>> Server's Module Magic Number: 20051115:3 >>> Server loaded: APR 1.5.0, APR-Util 1.5.3 >>> Compiled using: APR 1.2.7, APR-Util 1.2.7 >>> Architecture: 64-bit >>> Server MPM: Prefork >>> threaded: no >>> forked: yes (variable process count) >>> Server compiled with.... >>> -D APACHE_MPM_DIR="server/mpm/prefork" >>> -D APR_HAS_SENDFILE >>> -D APR_HAS_MMAP >>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>> -D APR_USE_SYSVSEM_SERIALIZE >>> -D APR_USE_PTHREAD_SERIALIZE >>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>> -D APR_HAS_OTHER_CHILD >>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>> -D DYNAMIC_MODULE_LIMIT=128 >>> -D HTTPD_ROOT="/etc/httpd" >>> -D SUEXEC_BIN="/usr/sbin/suexec" >>> -D DEFAULT_PIDLOG="run/httpd.pid" >>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>> -D DEFAULT_LOCKFILE="logs/accept.lock" >>> -D DEFAULT_ERRORLOG="logs/error_log" >>> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "modwsgi" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/modwsgi/DnvZ8lWokYs/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
