Can you add to the directory:
/opt/vcweb
two empty files called:
mod_wsgi.py
apache.py
and restart and look then to see if the debug logging outputs:
mod_wsgi (pid=%d): Imported 'mod_wsgi'.
mod_wsgi (pid=%d): Imported 'apache'.
If they are output, then it I know it is getting past that point.
I suspect it may be dying in the following code which was changed in the newer
version.
str = ap_get_server_description();
#if PY_MAJOR_VERSION >= 3
object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);
#else
object = PyString_FromString(str);
#endif
PyModule_AddObject(module, "description", object);
str = MPM_NAME;
#if PY_MAJOR_VERSION >= 3
object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);
#else
object = PyString_FromString(str);
#endif
PyModule_AddObject(module, "mpm_name", object);
str = ap_get_server_built();
#if PY_MAJOR_VERSION >= 3
object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);
#else
object = PyString_FromString(str);
#endif
PyModule_AddObject(module, "build_date", object);
Either the old Apache is returning NULL for some of the Apache calls, or Redhat
in modifying the server to suppress the server description, is causing a NULL
to be returned.
What you might do is check if your Apache configuration has:
ServerSignature Off
and set it back to:
ServerSignature On
and try again.
Even with that as Off it still should return an empty string and not a NULL
though. But then, for a standard Apache description, that flag should affect
the response from ap_get_server_description() anyway.
Graham
On 13/07/2014, at 2:17 PM, Allen Lee <[email protected]> wrote:
> I'm pretty sure it is a binary package that gets installed, though I'm not
> sure what version of httpd it is getting compiled against, I would presume it
> is the stock RHEL 5 httpd though. It is the latest version of httpd for RHEL
> 5. I've provisioned a new RHEL 7 instance and mod_wsgi is working fine there,
> so I may just chalk this up as the final straw for finally upgrading our VMs.
>
> The LogLevel is set to debug and I added the WSGIVerboseDebugging but didn't
> get any different output in the root error.log (constant segfaults from child
> pid) and vhost error.log..
>
> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Attach interpreter ''.
>
> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding '/opt/vcweb' to
> path.
>
> [Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding
> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.
>
> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Attach interpreter ''.
>
> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding '/opt/vcweb' to
> path.
>
> [Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding
> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.
>
> On Sunday, July 13, 2014 1:13:04 PM UTC-7, Graham Dumpleton wrote:
> Can you ensure that LogLevel directive in Apache is set to 'debug'.
>
> Also add the directive:
>
> WSGIVerboseDebugging On
>
> at global scope.
>
> The give me a snippet of the log messages from Apache/mod_wsgi that lead up
> to the segfault.
>
> I can possibly narrow it down from that.
>
> Graham
>
> On 13/07/2014, at 12:47 PM, Graham Dumpleton <[email protected]> wrote:
>
>> One quick question while I think about what changed between 4.1.3 and 4.2.4.
>>
>> When you install a newer version of mod_wsgi, does it get compiled from
>> source code against the specific Apache installation you are using, or are
>> you using a binary package?
>>
>> If using a binary package for mod_wsgi and it was compiled for a newer
>> version of Apache than you are actually using, that can be a source of
>> problems.
>>
>> Is the Apache version you are using (which is actually very old), the latest
>> Apache version available for your distribution?
>>
>> In general I would only recommend using at least Apache 2.2.18ish as minimum
>> as there were some issues in Apache which were fixed around Apache
>> 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the
>> reports seemed to vanish for newer versions. Exactly what the problem was I
>> am not sure, but believe it was related to SSL changes in Apache.
>>
>> Graham
>>
>> On 13/07/2014, at 12:23 PM, Allen Lee <[email protected]> wrote:
>>
>>> Hi Graham,
>>>
>>> Thank you for the fast response! I've been checking to see what happened
>>> prior and it looks the relevant packages that changed between the working
>>> -> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8 - the errors
>>> started after logrotate reloaded apache over the weekend. Responses to
>>> questions included inline:
>>>
>>>
>>> On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote:
>>> What did you do just prior to the weekend that changed anything?
>>>
>>> This must have been caused by a package update earlier in the weekj that
>>> manifested itself after httpd restarted at 4 AM this morning from a weekly
>>> logrotate. Possibly relevant entries from yum.log:
>>> Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64
>>> Jun 04 14:16:43 Installed:
>>> python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch
>>> Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch
>>> Jun 04 14:16:52 Erased: python27-distribute
>>> Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64
>>> Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64
>>> Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64
>>> Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch
>>> Jun 09 15:05:59 Erased: python27-virtualenv
>>> Jun 09 15:06:10 Erased: python-virtualenv
>>> Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64
>>> Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64
>>> Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64
>>> Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64
>>> Jun 20 14:53:49 Updated:
>>> python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch
>>> Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch
>>> Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64
>>> Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch
>>> Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64
>>> Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch
>>> Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64
>>> Jul 13 08:53:16 Installed:
>>> python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64
>>> Jul 13 09:06:15 Erased: subversion-python
>>>
>>>
>>> Did you upgrade the Python installation you are dependent on, but not
>>> recreate the Python virtual environment against the updated Python
>>> installation?
>>>
>>> I just checked this and noticed that my virtualenv Python was indeed
>>> different from the updated Python installation. I recreated the Python
>>> virtualenv using the updated system Python 2.7 (now there is no diff
>>> between virtualenv Python and system Python) and restarted the server but
>>> it still continually segfaults like it used to.
>>>
>>>
>>> Did you apply any update to the Apache version being used?
>>>
>>> Nope.
>>>
>>>
>>> Did you change the Apache configuration to load any additional Apache
>>> modules?
>>>
>>> Nope.
>>>
>>>
>>> Did you change the Apache configuration for any modules you were already
>>> using?
>>>
>>> Nope.
>>>
>>>
>>> How many Python installations exist on the system and where are they
>>> installed?
>>>
>>> There are actually 3 installs of Python on this antiquated VM, the stock
>>> RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu
>>> with it which I rather like), and python2.7 from the IUS repositories.
>>> Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo.
>>>
>>>
>>> What Python installation is your Python virtual environment based off?
>>>
>>> Python 2.7
>>>
>>>
>>> What Python installation was mod_wsgi compiled against?
>>>
>>> Python2.7, using the python27-mod_wsgi from the IUS repo as well.
>>>
>>>
>>> Is the mod_wsgi.so finding the correct Python shared library for the Python
>>> installation you expect it to use?
>>>
>>> http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library
>>>
>>> Yes, I did check this and it appears to be correct:
>>> # ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so
>>> linux-vdso.so.1 => (0x00007fff7e577000)
>>> libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0
>>> (0x00002b99ffd5d000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000)
>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000)
>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000)
>>> libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000)
>>> libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000)
>>> /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000)
>>>
>>>
>>> Are you using mod_wsgi daemon mode?
>>>
>>> Yes. I tried switching to embedded mode but it exhibits the same behavior.
>>> Here's the wsgi related config from my Apache vhost:
>>>
>>> WSGISocketPrefix run/wsgi
>>> WSGIPythonHome /opt/virtualenvs/vcweb
>>> <VirtualHost *:80>
>>> ...
>>> WSGIDaemonProcess vcweb user=apache group=commons threads=25
>>> python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages
>>> WSGIProcessGroup vcweb
>>> WSGIScriptAlias / /opt/vcweb/wsgi.py
>>> ...
>>> </VirtualHost>
>>>
>>>
>>> Is Apache segfaulting on startup in all child processes, or only for
>>> mod_wsgi daemon mode processes?
>>>
>>> It appears that it's only for mod_wsgi daemon mode processes, php and Java
>>> sites also served by the same webserver are still up.
>>>
>>>
>>> Is the segfault truly on startup, or only on the first request against a
>>> process?
>>>
>>> Yes, as soon as the server starts up my base httpd error.log starts
>>> spamming child pid segfaults (exit 11) and the vhost error.log spams
>>> messages along these lines:
>>>
>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter
>>> ''.
>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/vcweb'
>>> to path.
>>> [Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding
>>> '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.
>>>
>>>
>>> Sorry for all the questions, but there isn't a lot to go on at this point.
>>>
>>> The point of these questions is to prompt you to validate certain things
>>> about your installation and ensure they are correct, but responses to them
>>> will also help me to understand your setup and so be able to work out what
>>> the issue may be.
>>>
>>> Thank you for taking a look, much appreciated! I've been planning to
>>> upgrade these VMs to RHEL 7 this summer anyways so there's no need to
>>> plunge a lot of resources into this if you don't see anything immediately
>>> untoward.
>>>
>>>
>>> Graham
>>>
>>> On 13/07/2014, at 9:47 AM, Allen Lee <[email protected]> wrote:
>>>
>>>> (also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but
>>>> wasn't sure where the appropriate place for this is)
>>>>
>>>> I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS
>>>> repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running
>>>> within a virtualenv. I haven't had any problems until this weekend when
>>>> Apache has started to segfault constantly when trying to load the python
>>>> interpreter - messages like (child pid 12574 exit signal Segmentation
>>>> fault (11)).
>>>>
>>>> I've read through
>>>> https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes
>>>> and can confirm that:
>>>>
>>>> 1/ mod_python is not installed, and
>>>>
>>>> 2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running
>>>> 2.0.1 but I didn't encounter a segfault when running:
>>>>
>>>> (in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python
>>>> Python 2.7.7 (default, Jun 4 2014, 17:09:35)
>>>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>> >>> import pyexpat
>>>> >>> pyexpat.version_info
>>>> (1, 95, 8)
>>>> >>>
>>>>
>>>> I dumped the core and extracted the following from gdb but am at a loss at
>>>> this point, any additional pointers to try to identify the source of the
>>>> problem would be much appreciated!
>>>>
>>>> Results of gdb /usr/sbin/httpd /path/to/core.dump:
>>>> <snipped>
>>>> warning: no loadable sections found in added symbol-file system-supplied
>>>> DSO at 0x7fff1dffd000
>>>> Core was generated by `/usr/sbin/httpd'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6
>>>> (gdb) where
>>>> #0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6
>>>> #1 0x00002aed4a24e43b in PyString_FromString () from
>>>> /usr/lib64/libpython2.7.so.1.0
>>>> #2 0x00002aed49f8124b in ?? ()
>>>> #3 0x00002aed00005168 in ?? ()
>>>> #4 0x00002aed49f97190 in ?? ()
>>>> #5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0
>>>> #6 0x00000000ffffffff in ?? ()
>>>> #7 0x00002aed49f97190 in ?? ()
>>>> #8 0x00002aed3d826cf0 in ?? ()
>>>> #9 0x00002aed4a1a0b98 in ?? ()
>>>> #10 0x0000000000000001 in ?? ()
>>>> #11 0x00002aed3d826cf0 in ?? ()
>>>> #12 0x0000000000000000 in ?? ()
>>>> (gdb)
>>>>
>>>> # httpd -V
>>>> Server version: Apache/2.2.3
>>>> Server built: Mar 26 2014 08:47:55
>>>> Server's Module Magic Number: 20051115:3
>>>> Server loaded: APR 1.5.0, APR-Util 1.5.3
>>>> Compiled using: APR 1.2.7, APR-Util 1.2.7
>>>> Architecture: 64-bit
>>>> Server MPM: Prefork
>>>> threaded: no
>>>> forked: yes (variable process count)
>>>> Server compiled with....
>>>> -D APACHE_MPM_DIR="server/mpm/prefork"
>>>> -D APR_HAS_SENDFILE
>>>> -D APR_HAS_MMAP
>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>> -D APR_USE_SYSVSEM_SERIALIZE
>>>> -D APR_USE_PTHREAD_SERIALIZE
>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>> -D APR_HAS_OTHER_CHILD
>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>> -D DYNAMIC_MODULE_LIMIT=128
>>>> -D HTTPD_ROOT="/etc/httpd"
>>>> -D SUEXEC_BIN="/usr/sbin/suexec"
>>>> -D DEFAULT_PIDLOG="run/httpd.pid"
>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>> -D DEFAULT_LOCKFILE="logs/accept.lock"
>>>> -D DEFAULT_ERRORLOG="logs/error_log"
>>>> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>>>> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/modwsgi.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/modwsgi.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
