> 5) Anyone have any suggestions as to how to proceed with debugging > this, aside from rolling up my sleeves, and tucking in with GDB?
So, I rolled up my sleeves, and tucked in with GDB...The problem proved to be the issue documented here at the Django project under the section, "If you get a segmentation fault": http://docs.djangoproject.com/en/dev/howto/deployment/modpython/ httpd loads mod_php, which loads in a PHP extension containing MySQL bindings to the system's (single-threaded?) mysql library: /usr/lib64/ libmysqlclient.so.15. Because OpenSuSE is such an old system, this library implements the ABI for MySQL major number/version 15, or in other words, the libmysqlclient library uses version 15 of the data structures when talking to the server. Specifically, the MYSQL_FIELD structure occupies 120 bytes on a 64-bit system. mod_python--which can and does reside in the same httpd process as mod_php--loads up Reviewboard, Django, and the python bindings for MySQL, along with its own version of a MySQL library. However, the python-mysql package which shipped with OpenSuSE 10.3 is too old to work with the version of Django that Reviewboard 1.x line needs. As such, we have to build a new python-mysql from source, either using easy_install, or manually (which was easier to debug). However, on this particular system, in spite of the elderly/stale nature of the OS, the local IT group has installed a not-quite-as- creaky version of MySQL--specifically one which implements version 16 of the data structures--not version 15. (This change was introduced in mid 2007: http://lists.mysql.com/commits/31791 .) The result of this is that when we build python-mysql, we wind up building python bindings to a library which uses Version 16 of the data structures, and functions. This library's Version 16 definition of the MYSQL_FIELD struct is 128 bytes on a 64-bit system--8 more bytes than it was in version 15, thanks to an "extension" pointer, which should help preserve binary compatibility. The MySQL server is sufficiently backward-compatible to understand both version 15 and version 16 data structures and queries. The problem comes in when mod_python and mod_php are both dynamically loaded into the same httpd process. They both load in their own mysql libraries, and this causes some sort of problem. There must be some shared state or persistence which is causing the queries in the Version 16 library to receive Version 15 responses. Ultimately, we see a segmentation fault in strlen because (for some reason) we get a Version 15 response to a Version 16 mysql_real_query() call in MySQL-python. This result is an array of Version 15 MYSQL_FIELD structs (120-bytes apiece) being processed by Version 16 code, which expects each of these structs to occupy 128 bytes. This misalignment of structure boundaries eventually leads to a non-pointer being interpreted as a pointer which is fed to strlen, and when that happens, it all falls down. It seems like I have a few options here: 1) Use fastcgi instead of mod_python, and take my chances with memory leaks. 2) Rebuild mod_php with MySQL bindings that use the same Version16 library as mod_python does. Because the two modules should then use the same libraries, the problem should go away...in theory... 3) Try to rebuild MySQL-python with the version 15 library and header files. 4) Try something like mod_wsgi. Most likely I'm going to go with #1--it should work, and I have the existing system to use as an example for configuration, etc, which is a bonus. #2 means building more stuff at the risk of breaking things that are currently working, since we do serve PHP pages from this system. #3 seems more Frankenstein-y to me than I have time for (and the storm's not at its height, anyway...), and #4 is uninteresting since I'm currently waaayy behind schedule. Perhaps this advice in the Reviewboard Documentation should be amended, if not reconsidered: "If you’re using Apache, we highly recommend using mod_python, as fastcgi has been known to have memory leak issues with Review Board." 1) As of June 17th, mod_python is officially dead. http://blog.dscpl.com.au/2010/06/modpython-project-is-now-officially.html Shouldn't RB recommend an actively-maintained piece of software? While the code and documentation for mod_python are available, the FAQ, which allegedly describes this problem, and who knows, maybe even a few solutions--that'd be nice--in detail, is not. The mailing list archives for this project have been basically silent for the past few months as well. Decay has set in, there, and will only increase. 2) It would probably be useful to echo Django's warning about combining mod_python with mod_php, when using MySQL as a backend. Basically, if you're not serving PHP with this web server, you should be able to disable mod_php, and have mod_python work just fine. (That's what I should have done to quickly confirm that this was the problem.) Thanks, m@ -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~----------~----~----~----~------~----~------~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en