I'm glad you figured it out!
As far as what to do now, from what I've heard, the future is more
likely to involve wsgi than fastcgi. I think it's a little better with
regards to memory leaks, too.
On Tue, Oct 5, 2010 at 9:43 PM, mxbraun <matthew.br...@intel.com> wrote:
>> 5) Anyone have any suggestions as to how to proceed with debugging
>> this, aside from rolling up my sleeves, and tucking in with GDB?
> So, I rolled up my sleeves, and tucked in with GDB...The problem
> proved to be the issue documented here at the Django project under the
> section, "If you get a segmentation fault":
> httpd loads mod_php, which loads in a PHP extension containing MySQL
> bindings to the system's (single-threaded?) mysql library: /usr/lib64/
> libmysqlclient.so.15. Because OpenSuSE is such an old system, this
> library implements the ABI for MySQL major number/version 15, or in
> other words, the libmysqlclient library uses version 15 of the data
> structures when talking to the server. Specifically, the MYSQL_FIELD
> structure occupies 120 bytes on a 64-bit system.
> mod_python--which can and does reside in the same httpd process as
> mod_php--loads up Reviewboard, Django, and the python bindings for
> MySQL, along with its own version of a MySQL library. However, the
> python-mysql package which shipped with OpenSuSE 10.3 is too old to
> work with the version of Django that Reviewboard 1.x line needs. As
> such, we have to build a new python-mysql from source, either using
> easy_install, or manually (which was easier to debug).
> However, on this particular system, in spite of the elderly/stale
> nature of the OS, the local IT group has installed a not-quite-as-
> creaky version of MySQL--specifically one which implements version 16
> of the data structures--not version 15. (This change was introduced
> in mid 2007: http://lists.mysql.com/commits/31791 .) The result of
> this is that when we build python-mysql, we wind up building python
> bindings to a library which uses Version 16 of the data structures,
> and functions. This library's Version 16 definition of the
> MYSQL_FIELD struct is 128 bytes on a 64-bit system--8 more bytes than
> it was in version 15, thanks to an "extension" pointer, which should
> help preserve binary compatibility.
> The MySQL server is sufficiently backward-compatible to understand
> both version 15 and version 16 data structures and queries. The
> problem comes in when mod_python and mod_php are both dynamically
> loaded into the same httpd process. They both load in their own mysql
> libraries, and this causes some sort of problem. There must be some
> shared state or persistence which is causing the queries in the
> Version 16 library to receive Version 15 responses.
> Ultimately, we see a segmentation fault in strlen because (for some
> reason) we get a Version 15 response to a Version 16
> mysql_real_query() call in MySQL-python. This result is an array of
> Version 15 MYSQL_FIELD structs (120-bytes apiece) being processed by
> Version 16 code, which expects each of these structs to occupy 128
> bytes. This misalignment of structure boundaries eventually leads to a
> non-pointer being interpreted as a pointer which is fed to strlen, and
> when that happens, it all falls down.
> It seems like I have a few options here:
> 1) Use fastcgi instead of mod_python, and take my chances with memory
> 2) Rebuild mod_php with MySQL bindings that use the same Version16
> library as mod_python does. Because the two modules should then use
> the same libraries, the problem should go away...in theory...
> 3) Try to rebuild MySQL-python with the version 15 library and header
> 4) Try something like mod_wsgi.
> Most likely I'm going to go with #1--it should work, and I have the
> existing system to use as an example for configuration, etc, which is
> a bonus. #2 means building more stuff at the risk of breaking things
> that are currently working, since we do serve PHP pages from this
> system. #3 seems more Frankenstein-y to me than I have time for (and
> the storm's not at its height, anyway...), and #4 is uninteresting
> since I'm currently waaayy behind schedule.
> Perhaps this advice in the Reviewboard Documentation should be
> amended, if not reconsidered:
> "If you’re using Apache, we highly recommend using mod_python, as
> fastcgi has been known to have memory leak issues with Review Board."
> 1) As of June 17th, mod_python is officially dead.
> Shouldn't RB recommend an actively-maintained piece of software? While
> the code and documentation for mod_python are available, the FAQ,
> which allegedly describes this problem, and who knows, maybe even a
> few solutions--that'd be nice--in detail, is not. The mailing list
> archives for this project have been basically silent for the past few
> months as well. Decay has set in, there, and will only increase.
> 2) It would probably be useful to echo Django's warning about
> combining mod_python with mod_php, when using MySQL as a backend.
> Basically, if you're not serving PHP with this web server, you should
> be able to disable mod_php, and have mod_python work just fine.
> (That's what I should have done to quickly confirm that this was the
> Want to help the Review Board project? Donate today at
> Happy user? Let us know at http://www.reviewboard.org/users/
> To unsubscribe from this group, send email to
> For more options, visit this group at
Want to help the Review Board project? Donate today at
Happy user? Let us know at http://www.reviewboard.org/users/
To unsubscribe from this group, send email to
For more options, visit this group at