If you were using binary packages, compiling from source code may not be an option. If it is, can you try:
https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.tar.gz That addresses the issue with the missing ap_get_server_description() on older Apache versions. If you can't easily check then let me know and I will officially release it anyway. Graham On 13/07/2014, at 11:27 PM, Graham Dumpleton <[email protected]> wrote: > Arrrgggh. > > Changes with Apache 2.2.4 > > > *) The full server version information is now included in the error log at > > startup as well as server status reports, irrespective of the setting > > of the ServerTokens directive. ap_get_server_version() is now > > deprecated, and is replaced by ap_get_server_banner() and > > ap_get_server_description(). [Jeff Trawick] > > > So two possibilities here. > > Whoever compiled the mod_wsgi binary, if they did in fact compile against > Apache 2.2.3, didn't see the warning at compilation time about the prototype > for that function not being found and also did no testing of mod_wsgi against > that Apache version so as to notice that it would crash things on startup. > > Or, it was compiled against a newer Apache version which has that function > and did work okay. > > I will need to change the mod_wsgi code to check for the Apache module magic > number of 20051115.4 to see if the function is available in the version being > compiled against and if not use other means to generate the information, or > simply not include it. > > You might try and find out via the Redhat issue you created what version of > Apache the mod_wsgi binary was compiled against and point out that it has > been found that mod_wsgi will not work with Apache 2.2.3 and earlier and > mod_wsgi will need a fix to work with such old versions. > > I will try and create a new version of mod_wsgi tomorrow. > > Graham > > On 13/07/2014, at 11:08 PM, A Lee <[email protected]> wrote: > >> Hi Graham, >> >> I added the two files - it does emit "Imported 'mod_wsgi'." but *not* >> "Imported 'apache'." >> >> My apache config does indeed have >> >> ServerSignature On >> >> set properly. >> >> One thing strange I noticed related to ap_get_server_description - >> after I removed mod_perl v2.0.4-6 in an effort to remove >> unused/unneeded modules, httpd wouldn't even start if >> python27-mod_wsgi.so was loaded, complaining "Cannot load >> /etc/httpd/modules/python27-mod_wsgi.so into server: undefined symbol: >> ap_get_server_description". Removing the mod_wsgi load without >> mod_perl installed allowed httpd to start up normally. I'm not sure if >> this is related or not though... >> >> >> >> >> On Sun, Jul 13, 2014 at 8:39 PM, Graham Dumpleton >> <[email protected]> wrote: >>> Can you add to the directory: >>> >>> /opt/vcweb >>> >>> two empty files called: >>> >>> mod_wsgi.py >>> apache.py >>> >>> and restart and look then to see if the debug logging outputs: >>> >>> mod_wsgi (pid=%d): Imported 'mod_wsgi'. >>> mod_wsgi (pid=%d): Imported 'apache'. >>> >>> If they are output, then it I know it is getting past that point. >>> >>> I suspect it may be dying in the following code which was changed in the >>> newer version. >>> >>> str = ap_get_server_description(); >>> #if PY_MAJOR_VERSION >= 3 >>> object = PyUnicode_DecodeLatin1(str, strlen(str), NULL); >>> #else >>> object = PyString_FromString(str); >>> #endif >>> PyModule_AddObject(module, "description", object); >>> >>> str = MPM_NAME; >>> #if PY_MAJOR_VERSION >= 3 >>> object = PyUnicode_DecodeLatin1(str, strlen(str), NULL); >>> #else >>> object = PyString_FromString(str); >>> #endif >>> PyModule_AddObject(module, "mpm_name", object); >>> >>> str = ap_get_server_built(); >>> #if PY_MAJOR_VERSION >= 3 >>> object = PyUnicode_DecodeLatin1(str, strlen(str), NULL); >>> #else >>> object = PyString_FromString(str); >>> #endif >>> PyModule_AddObject(module, "build_date", object); >>> >>> Either the old Apache is returning NULL for some of the Apache calls, or >>> Redhat in modifying the server to suppress the server description, is >>> causing a NULL to be returned. >>> >>> What you might do is check if your Apache configuration has: >>> >>> ServerSignature Off >>> >>> and set it back to: >>> >>> ServerSignature On >>> >>> and try again. >>> >>> Even with that as Off it still should return an empty string and not a NULL >>> though. But then, for a standard Apache description, that flag should affect >>> the response from ap_get_server_description() anyway. >>> >>> Graham >>> >>> On 13/07/2014, at 2:17 PM, Allen Lee <[email protected]> wrote: >>> >>> I'm pretty sure it is a binary package that gets installed, though I'm not >>> sure what version of httpd it is getting compiled against, I would presume >>> it is the stock RHEL 5 httpd though. It is the latest version of httpd for >>> RHEL 5. I've provisioned a new RHEL 7 instance and mod_wsgi is working fine >>> there, so I may just chalk this up as the final straw for finally upgrading >>> our VMs. >>> >>> The LogLevel is set to debug and I added the WSGIVerboseDebugging but didn't >>> get any different output in the root error.log (constant segfaults from >>> child pid) and vhost error.log.. >>> >>> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Attach interpreter >>> ''. >>> >>> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding '/opt/vcweb' >>> to path. >>> >>> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding >>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. >>> >>> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Attach interpreter >>> ''. >>> >>> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding '/opt/vcweb' >>> to path. >>> >>> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding >>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. >>> >>> On Sunday, July 13, 2014 1:13:04 PM UTC-7, Graham Dumpleton wrote: >>>> >>>> Can you ensure that LogLevel directive in Apache is set to 'debug'. >>>> >>>> Also add the directive: >>>> >>>> WSGIVerboseDebugging On >>>> >>>> at global scope. >>>> >>>> The give me a snippet of the log messages from Apache/mod_wsgi that lead >>>> up to the segfault. >>>> >>>> I can possibly narrow it down from that. >>>> >>>> Graham >>>> >>>> On 13/07/2014, at 12:47 PM, Graham Dumpleton <[email protected]> >>>> wrote: >>>> >>>> One quick question while I think about what changed between 4.1.3 and >>>> 4.2.4. >>>> >>>> When you install a newer version of mod_wsgi, does it get compiled from >>>> source code against the specific Apache installation you are using, or are >>>> you using a binary package? >>>> >>>> If using a binary package for mod_wsgi and it was compiled for a newer >>>> version of Apache than you are actually using, that can be a source of >>>> problems. >>>> >>>> Is the Apache version you are using (which is actually very old), the >>>> latest Apache version available for your distribution? >>>> >>>> In general I would only recommend using at least Apache 2.2.18ish as >>>> minimum as there were some issues in Apache which were fixed around Apache >>>> 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the >>>> reports seemed to vanish for newer versions. Exactly what the problem was I >>>> am not sure, but believe it was related to SSL changes in Apache. >>>> >>>> Graham >>>> >>>> On 13/07/2014, at 12:23 PM, Allen Lee <[email protected]> wrote: >>>> >>>> Hi Graham, >>>> >>>> Thank you for the fast response! I've been checking to see what happened >>>> prior and it looks the relevant packages that changed between the working >>>> -> >>>> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors >>>> started after logrotate reloaded apache over the weekend. Responses to >>>> questions included inline: >>>> >>>> >>>> On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote: >>>>> >>>>> What did you do just prior to the weekend that changed anything? >>>> >>>> >>>> This must have been caused by a package update earlier in the weekj that >>>> manifested itself after httpd restarted at 4 AM this morning from a weekly >>>> logrotate. Possibly relevant entries from yum.log: >>>> Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64 >>>> Jun 04 14:16:43 Installed: >>>> python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch >>>> Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch >>>> Jun 04 14:16:52 Erased: python27-distribute >>>> Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64 >>>> Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64 >>>> Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64 >>>> Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch >>>> Jun 09 15:05:59 Erased: python27-virtualenv >>>> Jun 09 15:06:10 Erased: python-virtualenv >>>> Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64 >>>> Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64 >>>> Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64 >>>> Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64 >>>> Jun 20 14:53:49 Updated: >>>> python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch >>>> Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch >>>> Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64 >>>> Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch >>>> Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 >>>> Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch >>>> Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64 >>>> Jul 13 08:53:16 Installed: >>>> python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64 >>>> Jul 13 09:06:15 Erased: subversion-python >>>> >>>>> >>>>> >>>>> Did you upgrade the Python installation you are dependent on, but not >>>>> recreate the Python virtual environment against the updated Python >>>>> installation? >>>> >>>> >>>> I just checked this and noticed that my virtualenv Python was indeed >>>> different from the updated Python installation. I recreated the Python >>>> virtualenv using the updated system Python 2.7 (now there is no diff >>>> between >>>> virtualenv Python and system Python) and restarted the server but it still >>>> continually segfaults like it used to. >>>> >>>>> >>>>> >>>>> Did you apply any update to the Apache version being used? >>>> >>>> >>>> Nope. >>>> >>>>> >>>>> >>>>> Did you change the Apache configuration to load any additional Apache >>>>> modules? >>>> >>>> >>>> Nope. >>>> >>>>> >>>>> Did you change the Apache configuration for any modules you were already >>>>> using? >>>> >>>> >>>> Nope. >>>> >>>>> >>>>> How many Python installations exist on the system and where are they >>>>> installed? >>>> >>>> >>>> There are actually 3 installs of Python on this antiquated VM, the stock >>>> RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu >>>> with it which I rather like), and python2.7 from the IUS repositories. >>>> Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo. >>>> >>>>> >>>>> >>>>> What Python installation is your Python virtual environment based off? >>>> >>>> >>>> Python 2.7 >>>> >>>>> >>>>> >>>>> What Python installation was mod_wsgi compiled against? >>>> >>>> >>>> Python2.7, using the python27-mod_wsgi from the IUS repo as well. >>>> >>>>> >>>>> >>>>> Is the mod_wsgi.so finding the correct Python shared library for the >>>>> Python installation you expect it to use? >>>>> >>>>> >>>>> >>>>> http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library >>>> >>>> >>>> Yes, I did check this and it appears to be correct: >>>> # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so >>>> linux-vdso.so.1 => (0x00007fff7e577000) >>>> libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 >>>> (0x00002b99ffd5d000) >>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000) >>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000) >>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000) >>>> libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000) >>>> libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000) >>>> /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000) >>>> >>>>> >>>>> >>>>> Are you using mod_wsgi daemon mode? >>>> >>>> >>>> Yes. I tried switching to embedded mode but it exhibits the same behavior. >>>> Here's the wsgi related config from my Apache vhost: >>>> >>>> WSGISocketPrefix run/wsgi >>>> WSGIPythonHome /opt/virtualenvs/vcweb >>>> <VirtualHost *:80> >>>> ... >>>> WSGIDaemonProcess vcweb user=apache group=commons threads=25 >>>> python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages >>>> WSGIProcessGroup vcweb >>>> WSGIScriptAlias / /opt/vcweb/wsgi.py >>>> ... >>>> </VirtualHost> >>>> >>>>> >>>>> Is Apache segfaulting on startup in all child processes, or only for >>>>> mod_wsgi daemon mode processes? >>>> >>>> >>>> It appears that it's only for mod_wsgi daemon mode processes, php and Java >>>> sites also served by the same webserver are still up. >>>> >>>>> >>>>> Is the segfault truly on startup, or only on the first request against a >>>>> process? >>>> >>>> >>>> Yes, as soon as the server starts up my base httpd error.log starts >>>> spamming child pid segfaults (exit 11) and the vhost error.log spams >>>> messages along these lines: >>>> >>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter >>>> ''. >>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding >>>> '/opt/vcweb' to path. >>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding >>>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path. >>>> >>>>> >>>>> Sorry for all the questions, but there isn't a lot to go on at this >>>>> point. >>>>> >>>>> >>>>> The point of these questions is to prompt you to validate certain things >>>>> about your installation and ensure they are correct, but responses to them >>>>> will also help me to understand your setup and so be able to work out what >>>>> the issue may be. >>>> >>>> >>>> Thank you for taking a look, much appreciated! I've been planning to >>>> upgrade these VMs to RHEL 7 this summer anyways so there's no need to >>>> plunge >>>> a lot of resources into this if you don't see anything immediately >>>> untoward. >>>> >>>>> >>>>> >>>>> Graham >>>>> >>>>> On 13/07/2014, at 9:47 AM, Allen Lee <[email protected]> wrote: >>>>> >>>>> (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but >>>>> wasn't sure where the appropriate place for this is) >>>>> >>>>> I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS >>>>> repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running >>>>> within a virtualenv. I haven't had any problems until this weekend when >>>>> Apache has started to segfault constantly when trying to load the python >>>>> interpreter - messages like (child pid 12574 exit signal Segmentation >>>>> fault >>>>> (11)). >>>>> >>>>> I've read through >>>>> https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes >>>>> and can confirm that: >>>>> >>>>> 1/ mod_python is not installed, and >>>>> >>>>> 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running >>>>> 2.0.1 but I didn't encounter a segfault when running: >>>>> >>>>> (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python >>>>> Python 2.7.7 (default, Jun 4 2014, 17:09:35) >>>>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 >>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>>>>> import pyexpat >>>>>>>> pyexpat.version_info >>>>> (1, 95, 8) >>>>>>>> >>>>> >>>>> I dumped the core and extracted the following from gdb but am at a loss >>>>> at this point, any additional pointers to try to identify the source of >>>>> the >>>>> problem would be much appreciated! >>>>> >>>>> Results of gdb /usr/sbin/httpd /path/to/core.dump: >>>>> <snipped> >>>>> warning: no loadable sections found in added symbol-file system-supplied >>>>> DSO at 0x7fff1dffd000 >>>>> Core was generated by `/usr/sbin/httpd'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >>>>> (gdb) where >>>>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6 >>>>> #1 0x00002aed4a24e43b in PyString_FromString () from >>>>> /usr/lib64/libpython2.7.so.1.0 >>>>> #2 0x00002aed49f8124b in ?? () >>>>> #3 0x00002aed00005168 in ?? () >>>>> #4 0x00002aed49f97190 in ?? () >>>>> #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0 >>>>> #6 0x00000000ffffffff in ?? () >>>>> #7 0x00002aed49f97190 in ?? () >>>>> #8 0x00002aed3d826cf0 in ?? () >>>>> #9 0x00002aed4a1a0b98 in ?? () >>>>> #10 0x0000000000000001 in ?? () >>>>> #11 0x00002aed3d826cf0 in ?? () >>>>> #12 0x0000000000000000 in ?? () >>>>> (gdb) >>>>> >>>>> # httpd -V >>>>> Server version: Apache/2.2.3 >>>>> Server built: Mar 26 2014 08:47:55 >>>>> Server's Module Magic Number: 20051115:3 >>>>> Server loaded: APR 1.5.0, APR-Util 1.5.3 >>>>> Compiled using: APR 1.2.7, APR-Util 1.2.7 >>>>> Architecture: 64-bit >>>>> Server MPM: Prefork >>>>> threaded: no >>>>> forked: yes (variable process count) >>>>> Server compiled with.... >>>>> -D APACHE_MPM_DIR="server/mpm/prefork" >>>>> -D APR_HAS_SENDFILE >>>>> -D APR_HAS_MMAP >>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>>>> -D APR_USE_SYSVSEM_SERIALIZE >>>>> -D APR_USE_PTHREAD_SERIALIZE >>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>>>> -D APR_HAS_OTHER_CHILD >>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>>>> -D DYNAMIC_MODULE_LIMIT=128 >>>>> -D HTTPD_ROOT="/etc/httpd" >>>>> -D SUEXEC_BIN="/usr/sbin/suexec" >>>>> -D DEFAULT_PIDLOG="run/httpd.pid" >>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>>>> -D DEFAULT_LOCKFILE="logs/accept.lock" >>>>> -D DEFAULT_ERRORLOG="logs/error_log" >>>>> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>>>> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/modwsgi. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/modwsgi. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/DnvZ8lWokYs/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
