I'm glad you figured it out! As far as what to do now, from what I've heard, the future is more likely to involve wsgi than fastcgi. I think it's a little better with regards to memory leaks, too.
-David On Tue, Oct 5, 2010 at 9:43 PM, mxbraun <matthew.br...@intel.com> wrote: > >> 5) Anyone have any suggestions as to how to proceed with debugging >> this, aside from rolling up my sleeves, and tucking in with GDB? > > So, I rolled up my sleeves, and tucked in with GDB...The problem > proved to be the issue documented here at the Django project under the > section, "If you get a segmentation fault": > http://docs.djangoproject.com/en/dev/howto/deployment/modpython/ > > httpd loads mod_php, which loads in a PHP extension containing MySQL > bindings to the system's (single-threaded?) mysql library: /usr/lib64/ > libmysqlclient.so.15. Because OpenSuSE is such an old system, this > library implements the ABI for MySQL major number/version 15, or in > other words, the libmysqlclient library uses version 15 of the data > structures when talking to the server. Specifically, the MYSQL_FIELD > structure occupies 120 bytes on a 64-bit system. > > mod_python--which can and does reside in the same httpd process as > mod_php--loads up Reviewboard, Django, and the python bindings for > MySQL, along with its own version of a MySQL library. However, the > python-mysql package which shipped with OpenSuSE 10.3 is too old to > work with the version of Django that Reviewboard 1.x line needs. As > such, we have to build a new python-mysql from source, either using > easy_install, or manually (which was easier to debug). > > However, on this particular system, in spite of the elderly/stale > nature of the OS, the local IT group has installed a not-quite-as- > creaky version of MySQL--specifically one which implements version 16 > of the data structures--not version 15. (This change was introduced > in mid 2007: http://lists.mysql.com/commits/31791 .) The result of > this is that when we build python-mysql, we wind up building python > bindings to a library which uses Version 16 of the data structures, > and functions. This library's Version 16 definition of the > MYSQL_FIELD struct is 128 bytes on a 64-bit system--8 more bytes than > it was in version 15, thanks to an "extension" pointer, which should > help preserve binary compatibility. > > The MySQL server is sufficiently backward-compatible to understand > both version 15 and version 16 data structures and queries. The > problem comes in when mod_python and mod_php are both dynamically > loaded into the same httpd process. They both load in their own mysql > libraries, and this causes some sort of problem. There must be some > shared state or persistence which is causing the queries in the > Version 16 library to receive Version 15 responses. > > Ultimately, we see a segmentation fault in strlen because (for some > reason) we get a Version 15 response to a Version 16 > mysql_real_query() call in MySQL-python. This result is an array of > Version 15 MYSQL_FIELD structs (120-bytes apiece) being processed by > Version 16 code, which expects each of these structs to occupy 128 > bytes. This misalignment of structure boundaries eventually leads to a > non-pointer being interpreted as a pointer which is fed to strlen, and > when that happens, it all falls down. > > It seems like I have a few options here: > 1) Use fastcgi instead of mod_python, and take my chances with memory > leaks. > 2) Rebuild mod_php with MySQL bindings that use the same Version16 > library as mod_python does. Because the two modules should then use > the same libraries, the problem should go away...in theory... > 3) Try to rebuild MySQL-python with the version 15 library and header > files. > 4) Try something like mod_wsgi. > > Most likely I'm going to go with #1--it should work, and I have the > existing system to use as an example for configuration, etc, which is > a bonus. #2 means building more stuff at the risk of breaking things > that are currently working, since we do serve PHP pages from this > system. #3 seems more Frankenstein-y to me than I have time for (and > the storm's not at its height, anyway...), and #4 is uninteresting > since I'm currently waaayy behind schedule. > > Perhaps this advice in the Reviewboard Documentation should be > amended, if not reconsidered: > "If you’re using Apache, we highly recommend using mod_python, as > fastcgi has been known to have memory leak issues with Review Board." > 1) As of June 17th, mod_python is officially dead. > http://blog.dscpl.com.au/2010/06/modpython-project-is-now-officially.html > Shouldn't RB recommend an actively-maintained piece of software? While > the code and documentation for mod_python are available, the FAQ, > which allegedly describes this problem, and who knows, maybe even a > few solutions--that'd be nice--in detail, is not. The mailing list > archives for this project have been basically silent for the past few > months as well. Decay has set in, there, and will only increase. > 2) It would probably be useful to echo Django's warning about > combining mod_python with mod_php, when using MySQL as a backend. > Basically, if you're not serving PHP with this web server, you should > be able to disable mod_php, and have mod_python work just fine. > (That's what I should have done to quickly confirm that this was the > problem.) > > Thanks, > > m@ > > -- > Want to help the Review Board project? Donate today at > http://www.reviewboard.org/donate/ > Happy user? Let us know at http://www.reviewboard.org/users/ > -~----------~----~----~----~------~----~------~--~--- > To unsubscribe from this group, send email to > reviewboard+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/reviewboard?hl=en -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~----------~----~----~----~------~----~------~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en