Arrrgggh.
Changes with Apache 2.2.4
*) The full server version information is now included in the error log at
startup as well as server status reports, irrespective of the setting
of the ServerTokens directive. ap_get_server_version() is now
deprecated, and is replaced by ap_get_server_banner() and
ap_get_server_description(). [Jeff Trawick]
So two possibilities here.
Whoever compiled the mod_wsgi binary, if they did in fact compile against
Apache 2.2.3, didn't see the warning at compilation time about the prototype
for that function not being found and also did no testing of mod_wsgi against
that Apache version so as to notice that it would crash things on startup.
Or, it was compiled against a newer Apache version which has that function and
did work okay.
I will need to change the mod_wsgi code to check for the Apache module magic
number of 20051115.4 to see if the function is available in the version being
compiled against and if not use other means to generate the information, or
simply not include it.
You might try and find out via the Redhat issue you created what version of
Apache the mod_wsgi binary was compiled against and point out that it has been
found that mod_wsgi will not work with Apache 2.2.3 and earlier and mod_wsgi
will need a fix to work with such old versions.
I will try and create a new version of mod_wsgi tomorrow.
Graham
On 13/07/2014, at 11:08 PM, A Lee <[email protected]> wrote:
> Hi Graham,
>
> I added the two files - it does emit "Imported 'mod_wsgi'." but *not*
> "Imported 'apache'."
>
> My apache config does indeed have
>
> ServerSignature On
>
> set properly.
>
> One thing strange I noticed related to ap_get_server_description -
> after I removed mod_perl v2.0.4-6 in an effort to remove
> unused/unneeded modules, httpd wouldn't even start if
> python27-mod_wsgi.so was loaded, complaining "Cannot load
> /etc/httpd/modules/python27-mod_wsgi.so into server: undefined symbol:
> ap_get_server_description". Removing the mod_wsgi load without
> mod_perl installed allowed httpd to start up normally. I'm not sure if
> this is related or not though...
>
>
>
>
> On Sun, Jul 13, 2014 at 8:39 PM, Graham Dumpleton
> <[email protected]> wrote:
>> Can you add to the directory:
>>
>> /opt/vcweb
>>
>> two empty files called:
>>
>> mod_wsgi.py
>> apache.py
>>
>> and restart and look then to see if the debug logging outputs:
>>
>> mod_wsgi (pid=%d): Imported 'mod_wsgi'.
>> mod_wsgi (pid=%d): Imported 'apache'.
>>
>> If they are output, then it I know it is getting past that point.
>>
>> I suspect it may be dying in the following code which was changed in the
>> newer version.
>>
>> str = ap_get_server_description();
>> #if PY_MAJOR_VERSION >= 3
>> object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);
>> #else
>> object = PyString_FromString(str);
>> #endif
>> PyModule_AddObject(module, "description", object);
>>
>> str = MPM_NAME;
>> #if PY_MAJOR_VERSION >= 3
>> object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);
>> #else
>> object = PyString_FromString(str);
>> #endif
>> PyModule_AddObject(module, "mpm_name", object);
>>
>> str = ap_get_server_built();
>> #if PY_MAJOR_VERSION >= 3
>> object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);
>> #else
>> object = PyString_FromString(str);
>> #endif
>> PyModule_AddObject(module, "build_date", object);
>>
>> Either the old Apache is returning NULL for some of the Apache calls, or
>> Redhat in modifying the server to suppress the server description, is
>> causing a NULL to be returned.
>>
>> What you might do is check if your Apache configuration has:
>>
>> ServerSignature Off
>>
>> and set it back to:
>>
>> ServerSignature On
>>
>> and try again.
>>
>> Even with that as Off it still should return an empty string and not a NULL
>> though. But then, for a standard Apache description, that flag should affect
>> the response from ap_get_server_description() anyway.
>>
>> Graham
>>
>> On 13/07/2014, at 2:17 PM, Allen Lee <[email protected]> wrote:
>>
>> I'm pretty sure it is a binary package that gets installed, though I'm not
>> sure what version of httpd it is getting compiled against, I would presume
>> it is the stock RHEL 5 httpd though. It is the latest version of httpd for
>> RHEL 5. I've provisioned a new RHEL 7 instance and mod_wsgi is working fine
>> there, so I may just chalk this up as the final straw for finally upgrading
>> our VMs.
>>
>> The LogLevel is set to debug and I added the WSGIVerboseDebugging but didn't
>> get any different output in the root error.log (constant segfaults from
>> child pid) and vhost error.log..
>>
>> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Attach interpreter
>> ''.
>>
>> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding '/opt/vcweb'
>> to path.
>>
>> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding
>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.
>>
>> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Attach interpreter
>> ''.
>>
>> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding '/opt/vcweb'
>> to path.
>>
>> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding
>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.
>>
>> On Sunday, July 13, 2014 1:13:04 PM UTC-7, Graham Dumpleton wrote:
>>>
>>> Can you ensure that LogLevel directive in Apache is set to 'debug'.
>>>
>>> Also add the directive:
>>>
>>> WSGIVerboseDebugging On
>>>
>>> at global scope.
>>>
>>> The give me a snippet of the log messages from Apache/mod_wsgi that lead
>>> up to the segfault.
>>>
>>> I can possibly narrow it down from that.
>>>
>>> Graham
>>>
>>> On 13/07/2014, at 12:47 PM, Graham Dumpleton <[email protected]>
>>> wrote:
>>>
>>> One quick question while I think about what changed between 4.1.3 and
>>> 4.2.4.
>>>
>>> When you install a newer version of mod_wsgi, does it get compiled from
>>> source code against the specific Apache installation you are using, or are
>>> you using a binary package?
>>>
>>> If using a binary package for mod_wsgi and it was compiled for a newer
>>> version of Apache than you are actually using, that can be a source of
>>> problems.
>>>
>>> Is the Apache version you are using (which is actually very old), the
>>> latest Apache version available for your distribution?
>>>
>>> In general I would only recommend using at least Apache 2.2.18ish as
>>> minimum as there were some issues in Apache which were fixed around Apache
>>> 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the
>>> reports seemed to vanish for newer versions. Exactly what the problem was I
>>> am not sure, but believe it was related to SSL changes in Apache.
>>>
>>> Graham
>>>
>>> On 13/07/2014, at 12:23 PM, Allen Lee <[email protected]> wrote:
>>>
>>> Hi Graham,
>>>
>>> Thank you for the fast response! I've been checking to see what happened
>>> prior and it looks the relevant packages that changed between the working ->
>>> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors
>>> started after logrotate reloaded apache over the weekend. Responses to
>>> questions included inline:
>>>
>>>
>>> On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote:
>>>>
>>>> What did you do just prior to the weekend that changed anything?
>>>
>>>
>>> This must have been caused by a package update earlier in the weekj that
>>> manifested itself after httpd restarted at 4 AM this morning from a weekly
>>> logrotate. Possibly relevant entries from yum.log:
>>> Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64
>>> Jun 04 14:16:43 Installed:
>>> python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch
>>> Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch
>>> Jun 04 14:16:52 Erased: python27-distribute
>>> Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64
>>> Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64
>>> Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64
>>> Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch
>>> Jun 09 15:05:59 Erased: python27-virtualenv
>>> Jun 09 15:06:10 Erased: python-virtualenv
>>> Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64
>>> Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64
>>> Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64
>>> Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64
>>> Jun 20 14:53:49 Updated:
>>> python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch
>>> Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch
>>> Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64
>>> Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch
>>> Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64
>>> Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch
>>> Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64
>>> Jul 13 08:53:16 Installed:
>>> python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64
>>> Jul 13 09:06:15 Erased: subversion-python
>>>
>>>>
>>>>
>>>> Did you upgrade the Python installation you are dependent on, but not
>>>> recreate the Python virtual environment against the updated Python
>>>> installation?
>>>
>>>
>>> I just checked this and noticed that my virtualenv Python was indeed
>>> different from the updated Python installation. I recreated the Python
>>> virtualenv using the updated system Python 2.7 (now there is no diff between
>>> virtualenv Python and system Python) and restarted the server but it still
>>> continually segfaults like it used to.
>>>
>>>>
>>>>
>>>> Did you apply any update to the Apache version being used?
>>>
>>>
>>> Nope.
>>>
>>>>
>>>>
>>>> Did you change the Apache configuration to load any additional Apache
>>>> modules?
>>>
>>>
>>> Nope.
>>>
>>>>
>>>> Did you change the Apache configuration for any modules you were already
>>>> using?
>>>
>>>
>>> Nope.
>>>
>>>>
>>>> How many Python installations exist on the system and where are they
>>>> installed?
>>>
>>>
>>> There are actually 3 installs of Python on this antiquated VM, the stock
>>> RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu
>>> with it which I rather like), and python2.7 from the IUS repositories.
>>> Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo.
>>>
>>>>
>>>>
>>>> What Python installation is your Python virtual environment based off?
>>>
>>>
>>> Python 2.7
>>>
>>>>
>>>>
>>>> What Python installation was mod_wsgi compiled against?
>>>
>>>
>>> Python2.7, using the python27-mod_wsgi from the IUS repo as well.
>>>
>>>>
>>>>
>>>> Is the mod_wsgi.so finding the correct Python shared library for the
>>>> Python installation you expect it to use?
>>>>
>>>>
>>>>
>>>> http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library
>>>
>>>
>>> Yes, I did check this and it appears to be correct:
>>> # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so
>>> linux-vdso.so.1 => (0x00007fff7e577000)
>>> libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0
>>> (0x00002b99ffd5d000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000)
>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000)
>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000)
>>> libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000)
>>> libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000)
>>> /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000)
>>>
>>>>
>>>>
>>>> Are you using mod_wsgi daemon mode?
>>>
>>>
>>> Yes. I tried switching to embedded mode but it exhibits the same behavior.
>>> Here's the wsgi related config from my Apache vhost:
>>>
>>> WSGISocketPrefix run/wsgi
>>> WSGIPythonHome /opt/virtualenvs/vcweb
>>> <VirtualHost *:80>
>>> ...
>>> WSGIDaemonProcess vcweb user=apache group=commons threads=25
>>> python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages
>>> WSGIProcessGroup vcweb
>>> WSGIScriptAlias / /opt/vcweb/wsgi.py
>>> ...
>>> </VirtualHost>
>>>
>>>>
>>>> Is Apache segfaulting on startup in all child processes, or only for
>>>> mod_wsgi daemon mode processes?
>>>
>>>
>>> It appears that it's only for mod_wsgi daemon mode processes, php and Java
>>> sites also served by the same webserver are still up.
>>>
>>>>
>>>> Is the segfault truly on startup, or only on the first request against a
>>>> process?
>>>
>>>
>>> Yes, as soon as the server starts up my base httpd error.log starts
>>> spamming child pid segfaults (exit 11) and the vhost error.log spams
>>> messages along these lines:
>>>
>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter
>>> ''.
>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding
>>> '/opt/vcweb' to path.
>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding
>>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.
>>>
>>>>
>>>> Sorry for all the questions, but there isn't a lot to go on at this
>>>> point.
>>>>
>>>>
>>>> The point of these questions is to prompt you to validate certain things
>>>> about your installation and ensure they are correct, but responses to them
>>>> will also help me to understand your setup and so be able to work out what
>>>> the issue may be.
>>>
>>>
>>> Thank you for taking a look, much appreciated! I've been planning to
>>> upgrade these VMs to RHEL 7 this summer anyways so there's no need to plunge
>>> a lot of resources into this if you don't see anything immediately untoward.
>>>
>>>>
>>>>
>>>> Graham
>>>>
>>>> On 13/07/2014, at 9:47 AM, Allen Lee <[email protected]> wrote:
>>>>
>>>> (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but
>>>> wasn't sure where the appropriate place for this is)
>>>>
>>>> I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS
>>>> repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running
>>>> within a virtualenv. I haven't had any problems until this weekend when
>>>> Apache has started to segfault constantly when trying to load the python
>>>> interpreter - messages like (child pid 12574 exit signal Segmentation fault
>>>> (11)).
>>>>
>>>> I've read through
>>>> https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes
>>>> and can confirm that:
>>>>
>>>> 1/ mod_python is not installed, and
>>>>
>>>> 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running
>>>> 2.0.1 but I didn't encounter a segfault when running:
>>>>
>>>> (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python
>>>> Python 2.7.7 (default, Jun 4 2014, 17:09:35)
>>>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>> import pyexpat
>>>>>>> pyexpat.version_info
>>>> (1, 95, 8)
>>>>>>>
>>>>
>>>> I dumped the core and extracted the following from gdb but am at a loss
>>>> at this point, any additional pointers to try to identify the source of the
>>>> problem would be much appreciated!
>>>>
>>>> Results of gdb /usr/sbin/httpd /path/to/core.dump:
>>>> <snipped>
>>>> warning: no loadable sections found in added symbol-file system-supplied
>>>> DSO at 0x7fff1dffd000
>>>> Core was generated by `/usr/sbin/httpd'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6
>>>> (gdb) where
>>>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6
>>>> #1 0x00002aed4a24e43b in PyString_FromString () from
>>>> /usr/lib64/libpython2.7.so.1.0
>>>> #2 0x00002aed49f8124b in ?? ()
>>>> #3 0x00002aed00005168 in ?? ()
>>>> #4 0x00002aed49f97190 in ?? ()
>>>> #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0
>>>> #6 0x00000000ffffffff in ?? ()
>>>> #7 0x00002aed49f97190 in ?? ()
>>>> #8 0x00002aed3d826cf0 in ?? ()
>>>> #9 0x00002aed4a1a0b98 in ?? ()
>>>> #10 0x0000000000000001 in ?? ()
>>>> #11 0x00002aed3d826cf0 in ?? ()
>>>> #12 0x0000000000000000 in ?? ()
>>>> (gdb)
>>>>
>>>> # httpd -V
>>>> Server version: Apache/2.2.3
>>>> Server built: Mar 26 2014 08:47:55
>>>> Server's Module Magic Number: 20051115:3
>>>> Server loaded: APR 1.5.0, APR-Util 1.5.3
>>>> Compiled using: APR 1.2.7, APR-Util 1.2.7
>>>> Architecture: 64-bit
>>>> Server MPM: Prefork
>>>> threaded: no
>>>> forked: yes (variable process count)
>>>> Server compiled with....
>>>> -D APACHE_MPM_DIR="server/mpm/prefork"
>>>> -D APR_HAS_SENDFILE
>>>> -D APR_HAS_MMAP
>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>> -D APR_USE_SYSVSEM_SERIALIZE
>>>> -D APR_USE_PTHREAD_SERIALIZE
>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>> -D APR_HAS_OTHER_CHILD
>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>> -D DYNAMIC_MODULE_LIMIT=128
>>>> -D HTTPD_ROOT="/etc/httpd"
>>>> -D SUEXEC_BIN="/usr/sbin/suexec"
>>>> -D DEFAULT_PIDLOG="run/httpd.pid"
>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>> -D DEFAULT_LOCKFILE="logs/accept.lock"
>>>> -D DEFAULT_ERRORLOG="logs/error_log"
>>>> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>>>> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/modwsgi.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/modwsgi.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>>
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/modwsgi.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "modwsgi" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/modwsgi/DnvZ8lWokYs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/modwsgi.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
