Sorry for the late response.
a reminder:
I am using trac with wsgi, when trac comes under stress my apache server
locks up, once it's running out of worker threads.
You suggested that the problem may have been solved in later version of
sqlite. I am now running with 3.5.9 and upgraded to python 2.5 -
but the problem is still there.
in fact, I can easily reproduce it by using an http stress testing tool
(siege) on trac /timeline url.
it brings my entire apache server down in a matter of seconds.
I rebuild mod-wsgi with debug information, and this is the stack trace:
(gdb) bt
#0 0x00007fafff6fbbe1 in sem_wait () from /lib/libpthread.so.0
#1 0x00007faff6158c78 in PyThread_acquire_lock () from
/usr/lib/libpython2.5.so.1.0
#2 0x00007faff615df62 in ?? () from /usr/lib/libpython2.5.so.1.0
#3 0x00007faff612ffbd in PyEval_EvalFrameEx () from
/usr/lib/libpython2.5.so.1.0
#4 0x00007faff6131bdd in PyEval_EvalCodeEx () from
/usr/lib/libpython2.5.so.1.0
#5 0x00007faff61302a0 in PyEval_EvalFrameEx () from
/usr/lib/libpython2.5.so.1.0
#6 0x00007faff6131bdd in PyEval_EvalCodeEx () from
/usr/lib/libpython2.5.so.1.0
#7 0x00007faff61302a0 in PyEval_EvalFrameEx () from
/usr/lib/libpython2.5.so.1.0
#8 0x00007faff6131bdd in PyEval_EvalCodeEx () from
/usr/lib/libpython2.5.so.1.0
#9 0x00007faff61302a0 in PyEval_EvalFrameEx () from
/usr/lib/libpython2.5.so.1.0
#10 0x00007faff6131bdd in PyEval_EvalCodeEx () from
/usr/lib/libpython2.5.so.1.0
#11 0x00007faff61302a0 in PyEval_EvalFrameEx () from
/usr/lib/libpython2.5.so.1.0
#12 0x00007faff6131bdd in PyEval_EvalCodeEx () from
/usr/lib/libpython2.5.so.1.0
#13 0x00007faff61302a0 in PyEval_EvalFrameEx () from
/usr/lib/libpython2.5.so.1.0
#14 0x00007faff6131bdd in PyEval_EvalCodeEx () from
/usr/lib/libpython2.5.so.1.0
#15 0x00007faff60c855e in ?? () from /usr/lib/libpython2.5.so.1.0
#16 0x00007faff60a52a3 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0
#17 0x00007faff6129ff1 in PyEval_CallObjectWithKeywords () from
/usr/lib/libpython2.5.so.1.0
#18 0x00007faff63ee365 in Adapter_run () from
/usr/lib/apache2/modules/mod_wsgi.so
#19 0x00007faff63f263e in wsgi_execute_script () from
/usr/lib/apache2/modules/mod_wsgi.so
#20 0x00007faff63fb588 in wsgi_hook_daemon_handler () from
/usr/lib/apache2/modules/mod_wsgi.so
#21 0x00007faff63f79c9 in wsgi_process_socket () from
/usr/lib/apache2/modules/mod_wsgi.so
#22 0x00007faff63f7e17 in wsgi_daemon_worker () from
/usr/lib/apache2/modules/mod_wsgi.so
#23 0x00007faff63f7f16 in wsgi_daemon_thread () from
/usr/lib/apache2/modules/mod_wsgi.so
#24 0x00007fafff932171 in ?? () from /usr/lib/libapr-1.so.0
#25 0x00007fafff6f5fc7 in start_thread () from /lib/libpthread.so.0
#26 0x00007fafff46b7cd in clone () from /lib/libc.so.6
#27 0x0000000000000000 in ?? ()
The last thing in trac error log is this:
2008-12-13 08:38:02,070 Trac[main] ERROR: database is locked
Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/web/main.py",
line 423, in _dispatch_request
dispatcher.dispatch(req)
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/web/main.py",
line 173, in dispatch
chosen_handler)
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/web/main.py",
line 286, in _pre_process_request
chosen_handler = filter_.pre_process_request(req, chosen_handler)
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/versioncontrol/api.py",
line 86, in pre_process_request
self.get_repository(req.authname).sync()
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/versioncontrol/cache.py",
line 89, in sync
','.join(["'%s'" % key for key in CACHE_METADATA_KEYS]))
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/db/util.py",
line 51, in execute
return self.cursor.execute(sql)
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/db/sqlite_backend.py",
line 58, in execute
args or [])
File
"/usr/lib/python2.5/site-packages/Trac-0.11.1-py2.5.egg/trac/db/sqlite_backend.py",
line 50, in _rollback_on_error
return function(self, *args, **kwargs)
OperationalError: database is locked
Graham Dumpleton wrote:
> >
> > On Oct 7, 5:55 pm, Omry Yadan <[email protected]> wrote:
> >
>> >> Graham Dumpleton wrote:
>> >>
>>>>>> >>>>>> I have a problem apache workers running out due to trac
misbehavior.
>>>>>> >>>>>> if anything, I will reduce this number to something that
will not take
>>>>>> >>>>>> down the entire apache server whenever trac is acting up.
>>>>>> >>>>>>
>>>>> >>>>> What database adapter are you using? If using an older
version of it
>>>>> >>>>> than newest available, make sure you update it. Old versions of
>>>>> >>>>> psycopg2 have problems.
>>>>> >>>>>
>>>> >>>> I am using sqlite 3.3.8 with python-pysqlite2 2.3.2
>>>> >>>>
>>> >>> Which is not the latest version of either package, which means any
>>> >>> problem could well have been fixed long ago.
>>> >>>
>> >> changed WSGIDaemonProcess line to:
>> >> WSGIDaemonProcess firestats.cc user=omry group=omry threads=25
>> >> maximum-requests=100 inactivity-timeout=120
>> >>
>> >> but it froze again (this time it took longer to happen, not sure
if it's
>> >> related).
>> >> didn't had the chance to upgrade sqlite yet.
>> >>
>> >> my server froze again, 155 apache processes.
>> >>
>> >> this is the stack trace on one :
>> >>
>> >> (gdb) thread apply all bt
>> >>
>> >> Thread 1 (Thread 140422165649104 (LWP 28894)):
>> >> ...
>> >> #15 0x0000000000420e70 in main ()
>> >> #0 0x00007fb6942a7182 in __read_nocancel () from /lib/libpthread.so.0
>> >>
>> >> Here is the stacktrace of another:
>> >>
>> >> #0 0x00007fb6942a7472 in __connect_nocancel () from
/lib/libpthread.so.0
>> >> #1 0x00007fb690210591 in ?? () from
/usr/lib/apache2/modules/mod_wsgi.so
>> >> #2 ...
>> >> #12 0x0000000000420e70 in main ()
>> >>
>> >> anything useful here?
>> >>
> >
> > No, for a couple of reasons.
> >
> > The first is that debug information not available in mod_wsgi as not
> > compiled with debugging. Thus ensure that -g option is included when
> > compiling mod_wsgi.
> >
> > Second is that actually want the tracebacks for mod_wsgi daemon
> > process and not Apache worker processes. Use option:
> >
> > display-name=%{GROUP}88
> >
> > with WSGIDaemonProcess. Use 'ps' command to identify the mod_wsgi
> > daemon process and attach gdb to it, then get tracebacks.
> >
> > Reason for getting tracebacks for daemon processes is to ascertain if
> > threads there are stuck or not. If not, then it is an Apache
> > configuration issue which is most likely nothing to do with mod_wsgi.78
> >
> > BTW, when using mod_wsgi daemon mode, if not running PHP in Apache,
> > then you are better off running Apacher worker MPM rather than
> > prefork. That way number of processes less and overall Apache will use
> > less memory
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac
Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/trac-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
