Hi folks. Short description: Trying to set up new install of RB 1.0.9 on SLES 10.3. After creating a reviewboard, httpd is crashing, apparently during user authentication. I doubt the problem is RB directly--more likely an issue with python interfacing to MySQL. Anyone seen this, or know how to debug this?
Long description: We've got a production-ish system running a beta version of RB, with fastcgi, and SQLlite--yes, pretty much like the current docs advise *not* to. We've lived with this for too long, and there are a lot of complaints about timeouts, so I volunteered to move us up to a more recent version. (We won't talk about legacy data, migration, etc. That's a different story, which may not have a happy ending. The easiest ending may simply be just leaving the old system sitting around. This message is not about that issue.) So about two weeks ago, (before 1.5 was finalized) I started bringing up 1.0.9 on a new system. The specs: SELS 10.3 Apache 2.2 MySQL 5.1.41 In my first stab at this, I figured, "the system software's a bit old...I don't know how well it's going to work with newer software. I should build everything I can." So, I built the latest versions of needed software, including: Python 2.6.7, Tcl/Tk 8.5, PIL 1.7, memcached, python_memcached, mysql-python, subversion, pysvn I followed the install directions in the RB documentation, and easy_install'ed whatever versions of django, djblets, and the rest, all according to whatever easy_install decided to fetch. I created a reviewboard, and aimed my browser at the result. After climbing the Apache and MySQL learning curves to handle a non-standard MySQL socket name (fixed by patching django), server 403 errors (default out-of-the-box permissions for apache didn't allow access; needed FollowSymLinks in vhost config since RB egg files and www dirs are on different drives) and a server that refused to come up (unneeded subversion apache modules used a different version of the apache runtime than the server, resulting in crashes), I finally had a running server, and my browser proudly displayed the RB login page. Then I tried logging in as administrator, and httpd died with a segfault. :-( As a matter of fact, trying to log in as anyone (valid or not) yielded the same. After clicking the "log in" button, the httpd logs don't show a new URL request, so it seems that the crash is happening when RB is trying to authenticate the user--before RB can figure out where to go next. From the looks of it, it's going into MySQL to get that info. Now clearly, this doesn't seem likely to be an RB problem per se--RB is written in python, and if there are problems with it, we should see a python backtrace--not a segfault in httpd. We'll get to the details in the segfault in a moment. So, my immediate thought was that in my contempt for old software which lead me to rebuild the world (OpenSuSE doesn't support 10.3 any more, although commercial vendors do), I'd screwed up, and should have set my sights lower, using whatever packages shipped with the system. So I moved aside all of the stuff I'd built in /usr/local, and started again. The system ships with python 2.4, and that can run RB. I installed the appropriate RPMs, and went from there. I installed RB again, in the native python 2.4 site-packages directory. I had a problem here, because the "native" python-mysql module was too old for django to handle, so I had to find an update for that. Reinstall, rebuild, create a new reviewboard. Once again, I aimed the browser at it, and got the intro page. And once again, logging in gave me the same segfault in httpd. So the problem *wasn't* with the software I'd built. But I'm back where I'm started--or, actually a little worse. When I built the world, I had debug info for the python interpreter in GDB. With the new ones, I just get frame information:-( So about that segfault. I attached GDB to httpd and got the following for the top 6 frames of the backtrace. This is from the "I built the world" version, although the native one is identical as well--just without most source file names: Program received signal SIGSEGV, Segmentation fault. 0x00002aae825f6b30 in strlen () from /lib64/libc.so.6 (gdb) where #0 0x00002aae825f6b30 in strlen () from /lib64/libc.so.6 #1 0x00002aae850d4033 in do_mkvalue (p_format=0x7fff294bfe48, p_va=0x7fff294bfe30, flags=0) at Python/modsupport.c:418 #2 0x00002aae850d429d in do_mktuple (p_format=0x7fff294bfe48, p_va=0x7fff294bfe30, endchar=41, n=7, flags=0) at Python/ modsupport.c:267 #3 0x00002aae850d3c8c in do_mkvalue (p_format=0x7fff294bfe48, p_va=0x7fff294bfe30, flags=0) at Python/modsupport.c:297 #4 0x00002aae850d445c in va_build_value (format=<value optimized out>, va=0x7fff294bfe60, flags=-7) at Python/modsupport.c:536 #5 0x00002aae850d47df in Py_BuildValue ( format=0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>) at Python/modsupport.c:484 #6 0x00002aae8ab9284f in _mysql_ResultObject_describe ( self=<value optimized out>, args=<value optimized out>) at _mysql.c:1178 (plus 70 more frames inside of the python interpreter, mod_python.so, and httpd2-prefork) I have run a trivial test with python-mysql, and I can do things like retrieve the version, and retrieve a table name. The code which is crashing here is in python-mysql. The function _mysql_ResultObject_describe documents itself as: "Returns the sequence of 7-tuples required by the DB-API for\n\ the Cursor.description attribute.\n\ So, on to the questions: 1) Has anyone seen this problem? 2) Got a quick and/or easy solution? ;-) 3) Okay, how about a slow/difficult solution? 4) Is there any reason to think that updating to 1.5 would solve the problem? (The issue seems to be in software "under" Reviewboard, and not RB itself, so I'm skeptical that this would help--but I'm willing to listen.) 5) Anyone have any suggestions as to how to proceed with debugging this, aside from rolling up my sleeves, and tucking in with GDB? Thanks for any help you can offer. m@ -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~----------~----~----~----~------~----~------~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en