> 5) Anyone have any suggestions as to how to proceed with debugging
> this, aside from rolling up my sleeves, and tucking in with GDB?

So, I rolled up my sleeves, and tucked in with GDB...The problem
proved to be the issue documented here at the Django project under the
section, "If you get a segmentation fault":

httpd loads mod_php, which loads in a PHP extension containing MySQL
bindings to the system's (single-threaded?) mysql library: /usr/lib64/
libmysqlclient.so.15.  Because OpenSuSE is such an old system, this
library implements the ABI for MySQL major number/version 15, or in
other words, the libmysqlclient library uses version 15 of the data
structures when talking to the server.  Specifically, the MYSQL_FIELD
structure occupies 120 bytes on a 64-bit system.

mod_python--which can and does reside in the same httpd process as
mod_php--loads up Reviewboard, Django, and the python bindings for
MySQL, along with its own version of a MySQL library.  However, the
python-mysql package which shipped with OpenSuSE 10.3 is too old to
work with the version of Django that Reviewboard 1.x line needs.  As
such, we have to build a new python-mysql from source, either using
easy_install, or manually (which was easier to debug).

However, on this particular system, in spite of the elderly/stale
nature of the OS, the local IT group has installed a not-quite-as-
creaky version of MySQL--specifically one which implements version 16
of the data structures--not version 15.  (This change was introduced
in mid 2007: http://lists.mysql.com/commits/31791 .) The result of
this is that when we build python-mysql, we wind up building python
bindings to a library which uses Version 16 of the data structures,
and functions.  This library's Version 16 definition of the
MYSQL_FIELD struct is 128 bytes on a 64-bit system--8 more bytes than
it was in version 15, thanks to an "extension" pointer, which should
help preserve binary compatibility.

The MySQL server is sufficiently backward-compatible  to understand
both version 15 and version 16 data structures and queries.  The
problem comes in when mod_python and mod_php are both dynamically
loaded into the same httpd process.  They both load in their own mysql
libraries, and this causes some sort of problem.  There must be some
shared state or persistence which is causing the queries in the
Version 16 library to receive Version 15 responses.

Ultimately, we see a segmentation fault in strlen because (for some
reason) we get a Version 15 response to a Version 16
mysql_real_query() call in MySQL-python.  This result is an array of
Version 15 MYSQL_FIELD structs (120-bytes apiece) being processed by
Version 16 code, which expects each of these structs to occupy 128
bytes. This misalignment of structure boundaries eventually leads to a
non-pointer being interpreted as a pointer which is fed to strlen, and
when that happens, it all falls down.

It seems like I have a few options here:
1) Use fastcgi instead of mod_python, and take my chances with memory
2) Rebuild mod_php with MySQL bindings that use the same Version16
library as mod_python does.  Because the two modules should then use
the same libraries, the problem should go away...in theory...
3) Try to rebuild MySQL-python with the version 15 library and header
4) Try something like mod_wsgi.

Most likely I'm going to go with #1--it should work, and I have the
existing system to use as an example for configuration, etc, which is
a bonus. #2 means building more stuff at the risk of breaking things
that are currently working, since we do serve PHP pages from this
system. #3 seems more Frankenstein-y to me than I have time for (and
the storm's not at its height, anyway...), and #4 is uninteresting
since I'm currently waaayy behind schedule.

Perhaps this advice in the Reviewboard Documentation should be
amended, if not reconsidered:
        "If you’re using Apache, we highly recommend using mod_python, as
fastcgi has been known to have memory leak issues with Review Board."
1) As of June 17th, mod_python is officially dead.
Shouldn't RB recommend an actively-maintained piece of software? While
the code and documentation for mod_python are available, the FAQ,
which allegedly describes this problem, and who knows, maybe even a
few solutions--that'd be nice--in detail, is not. The mailing list
archives for this project have been basically silent for the past few
months as well.  Decay has set in, there, and will only increase.
2) It would probably be useful to echo Django's warning about
combining mod_python with mod_php, when using MySQL as a backend.
Basically, if you're not serving PHP with this web server, you should
be able to disable mod_php, and have mod_python work just fine.
(That's what I should have done to quickly confirm that this was the



Want to help the Review Board project? Donate today at 
Happy user? Let us know at http://www.reviewboard.org/users/
To unsubscribe from this group, send email to 
For more options, visit this group at 

Reply via email to