Hi folks.

Short description:  Trying to set up new install of RB 1.0.9 on SLES
10.3. After creating a reviewboard, httpd is crashing, apparently
during user authentication.  I doubt the problem is RB directly--more
likely an issue with python interfacing to MySQL.  Anyone seen this,
or know how to debug this?


Long description:

We've got a production-ish system running a beta version of RB, with
fastcgi, and SQLlite--yes, pretty much like the current docs advise
*not* to.  We've lived with this for too long, and there are a lot of
complaints about timeouts, so I volunteered to move us up to a more
recent version.  (We won't talk about legacy data, migration, etc.
That's a different story, which may not have a happy ending.  The
easiest ending may simply be just leaving the old system sitting
around.  This message is not about that issue.)

So about two weeks ago, (before 1.5 was finalized) I started bringing
up 1.0.9 on a new system.  The specs:
   SELS 10.3
   Apache 2.2
   MySQL 5.1.41

In my first stab at this, I figured, "the system software's a bit
old...I don't know how well it's going to work with newer software.  I
should build everything I can."  So, I built the latest versions of
needed software, including:
   Python 2.6.7, Tcl/Tk 8.5, PIL 1.7, memcached, python_memcached,
   mysql-python, subversion, pysvn

I followed the install directions in the RB documentation, and
easy_install'ed whatever versions of django, djblets, and the rest,
all according to whatever easy_install decided to fetch.

I created a reviewboard, and aimed my browser at the result.  After
climbing the Apache and MySQL learning curves to handle a non-standard
MySQL socket name (fixed by patching django), server 403 errors
(default out-of-the-box permissions for apache didn't allow access;
needed FollowSymLinks in vhost config since RB egg files and www dirs
are on different drives) and a server that refused to come up
(unneeded subversion apache modules used a different version of the
apache runtime than the server, resulting in crashes), I finally had a
running server, and my browser proudly displayed the RB login page.

Then I tried logging in as administrator, and httpd died with a
segfault.  :-(

As a matter of fact, trying to log in as anyone (valid or not) yielded
the same.  After clicking the "log in" button, the httpd logs don't
show a new URL request, so it seems that the crash is happening when
RB is trying to authenticate the user--before RB can figure out where
to go next.  From the looks of it, it's going into MySQL to get that
info.

Now clearly, this doesn't seem likely to be an RB problem per se--RB
is written in python, and if there are problems with it, we should see
a python backtrace--not a segfault in httpd.   We'll get to the
details in the segfault in a moment.

So, my immediate thought was that in my contempt for old software
which lead me to rebuild the world (OpenSuSE doesn't support 10.3 any
more, although commercial vendors do), I'd screwed up, and should have
set my sights lower, using whatever packages shipped with the system.
So I moved aside all of the stuff I'd built in /usr/local, and started
again.  The system ships with python 2.4, and that can run RB.  I
installed the appropriate RPMs, and went from there.  I installed RB
again, in the native python 2.4 site-packages directory.  I had a
problem here, because the "native" python-mysql module was too old for
django to handle, so I had to find an update for that.  Reinstall,
rebuild, create a new reviewboard.

Once again, I aimed the browser at it, and got the intro page.  And
once again, logging in gave me the same segfault in httpd.  So the
problem *wasn't* with the software I'd built. But I'm back where I'm
started--or, actually a little worse.  When I built the world, I had
debug info for the python interpreter in GDB. With the new ones, I
just get frame information:-(

So about that segfault.  I attached GDB to httpd and got the following
for the top 6 frames of the backtrace.  This is from the "I built the
world" version, although the native one is identical as well--just
without most source file names:
Program received signal SIGSEGV, Segmentation fault.
0x00002aae825f6b30 in strlen () from /lib64/libc.so.6
(gdb) where
#0  0x00002aae825f6b30 in strlen () from /lib64/libc.so.6
#1  0x00002aae850d4033 in do_mkvalue (p_format=0x7fff294bfe48,
    p_va=0x7fff294bfe30, flags=0) at Python/modsupport.c:418
#2  0x00002aae850d429d in do_mktuple (p_format=0x7fff294bfe48,
    p_va=0x7fff294bfe30, endchar=41, n=7, flags=0) at Python/
modsupport.c:267
#3  0x00002aae850d3c8c in do_mkvalue (p_format=0x7fff294bfe48,
    p_va=0x7fff294bfe30, flags=0) at Python/modsupport.c:297
#4  0x00002aae850d445c in va_build_value (format=<value optimized
out>,
    va=0x7fff294bfe60, flags=-7) at Python/modsupport.c:536
#5  0x00002aae850d47df in Py_BuildValue (
    format=0xffffffffffffffff <Address 0xffffffffffffffff out of
bounds>)
    at Python/modsupport.c:484
#6  0x00002aae8ab9284f in _mysql_ResultObject_describe (
    self=<value optimized out>, args=<value optimized out>) at
_mysql.c:1178
(plus 70 more frames inside of the python interpreter, mod_python.so,
and httpd2-prefork)

I have run a trivial test with python-mysql, and I can do things like
retrieve the version, and retrieve a table name.  The code which is
crashing here is in python-mysql.  The function
_mysql_ResultObject_describe documents itself as:
    "Returns the sequence of 7-tuples required by the DB-API for\n\
      the Cursor.description attribute.\n\

So, on to the questions:
1) Has anyone seen this problem?
2) Got a quick and/or easy solution? ;-)
3) Okay, how about a slow/difficult solution?
4) Is there any reason to think that updating to 1.5 would solve the
problem?  (The issue seems to be in software "under" Reviewboard, and
not RB itself, so I'm skeptical that this
would help--but I'm willing to listen.)
5) Anyone have any suggestions as to how to proceed with debugging
this, aside from rolling up my sleeves, and tucking in with GDB?

Thanks for any help you can offer.

m@


-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~----------~----~----~----~------~----~------~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en

Reply via email to