Hi all, I am starting a second thread with the same name, but it's not exactly the same case: I have trac accessed via mod_wsgi, and pretty much every day - my apache gets stuck and being automatically restarted by monit after a few minutes.
I am using a script that dumps the output of this command into a log file every 10 seconds, to get a snapshot of what apache was doing when it got stuck. links -dump http://localhost/server-status | grep ^[0-9]|awk 'BEGIN {print "Seconds, PID, State, IP, Domain, TYPE, URL\n--"} $4 !~ /[GCRK_.]/ {print $6, $2, $4, $11, $12, $13 " " $14|"sort -n"}' the attached file is a snapshot from the latest incident. last time I had the issue, the problem was with sqlite connection pooling. this is *no longer the issue as I am now using mysql database backend*. the crash.log file show that some processes have been running for nearly 1000 seconds, this conflicts with the apache configuration: --------------------- WSGIDaemonProcess firestats.cc user=omry group=omry threads=50 maximum-requests=500 inactivity-timeout=60 display-name=%{GROUP} deadlock-timeout=60 <VirtualHost *:80> WSGIProcessGroup firestats.cc WSGIApplicationGroup %{GLOBAL} ServerName firestats.cc ServerAlias www.firestats.cc ServerAdmin omry ErrorLog /home/omry/logs/firestats.cc/error.log WSGIScriptAlias / /home/omry/www/trac/trac.wsgi SetEnv trac.env_path /home/omry/www/trac/firestats.cc RewriteEngine on RewriteRule ^/robots.txt /home/omry/www/trac/robots.txt [L] </VirtualHost> --------------------- WSGIDaemonProcess is set to have inactivity and deadlock timeouts of 60 seconds, so how can a process possibly run for 1000 seconds? using siege on the timeline URL gets the server to a bad spot (with 45 concurrent clients, I can bring my server down in seconds). I tried to inspect the apache backtrace using gdb , I get this in the first thread: (gdb) bt #0 0x00007f3fcd588b66 in poll () from /lib/libc.so.6 #1 0x00007f3fcda54e95 in apr_poll () from /usr/lib/libapr-1.so.0 #2 0x00007f3fc430c53c in wsgi_daemon_main () from /usr/lib/apache2/modules/mod_wsgi.so #3 0x00007f3fc430cd39 in wsgi_start_process () from /usr/lib/apache2/modules/mod_wsgi.so #4 0x00007f3fc430d339 in wsgi_start_daemons () from /usr/lib/apache2/modules/mod_wsgi.so #5 0x00007f3fc430f640 in wsgi_hook_init () from /usr/lib/apache2/modules/mod_wsgi.so #6 0x0000000000438cf4 in ap_run_post_config () #7 0x0000000000425bbc in main () but after switching to other threads I get this: #0 0x00007f3fcd821bd1 in sem_wait () from /lib/libpthread.so.0 #1 0x00007f3fc406bbc8 in PyThread_acquire_lock (lock=0xcbaf30, waitflag=128) at ../Python/thread_pthread.h:349 #2 0x00007f3fc4070ec2 in lock_PyThread_acquire_lock (self=0xc3f2a0, args=<value optimized out>) at ../Modules/threadmodule.c:46 #3 0x00007f3fc4042fcd in PyEval_EvalFrameEx (f=0x2a7cec0, throwflag=<value optimized out>) at ../Python/ceval.c:3575 #4 0x00007f3fc4044bed in PyEval_EvalCodeEx (co=0x1176cd8, globals=<value optimized out>, locals=<value optimized out>, args=0x2, argcount=1, kws=0x2abc5d8, kwcount=0, defs=0x11854a8, defcount=1, closure=0x0) at ../Python/ceval.c:2838 .... #141 0x00007f3fc4044375 in PyEval_EvalFrameEx (f=0x3e303c0, throwflag=<value optimized out>) at ../Python/ceval.c:3661 #142 0x00007f3fc4044375 in PyEval_EvalFrameEx (f=0x3ece7d0, throwflag=<value optimized out>) at ../Python/ceval.c:3661 #143 0x00007f3fc4044bed in PyEval_EvalCodeEx (co=0xe1e210, globals=<value optimized out>, locals=<value optimized out>, args=0x35290b0, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2838 #144 0x00007f3fc3fdb55e in function_call (func=0x1418938, arg=0x3529098, kw=0x0) at ../Objects/funcobject.c:517 #145 0x00007f3fc3fb82a3 in PyObject_Call (func=0xcbaf30, arg=0x80, kw=0x0) at ../Objects/abstract.c:1861 #146 0x00007f3fc403d001 in PyEval_CallObjectWithKeywords (func=0x1418938, arg=0x3529098, kw=0x0) at ../Python/ceval.c:3444 #147 0x00007f3fc430183c in Adapter_run () from /usr/lib/apache2/modules/mod_wsgi.so #148 0x00007f3fc430632d in wsgi_execute_script () from /usr/lib/apache2/modules/mod_wsgi.so #149 0x00007f3fc430f3cd in wsgi_hook_daemon_handler () from /usr/lib/apache2/modules/mod_wsgi.so #150 0x00007f3fc430b6ac in wsgi_process_socket () from /usr/lib/apache2/modules/mod_wsgi.so #151 0x00007f3fc430bafa in wsgi_daemon_worker () from /usr/lib/apache2/modules/mod_wsgi.so #152 0x00007f3fc430bbf9 in wsgi_daemon_thread () from /usr/lib/apache2/modules/mod_wsgi.so which does not look too helpful to my untrained eye. I am using the python 2.5 gdbinit extentions (cp /usr/share/doc/python2.5/gdbinit ~.gdbinit), and if I am not mistaken - I was supposed to get python stack trace and not python interpreter stack trace. any help will be appreciated. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Trac Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en -~----------~----~----~----~------~----~------~--~---
0 15408 W 127.0.0.1 flux.firefang.net GET /server-status 9 15407 W 72.30.161.232 firestats.cc GET /ticket/112?format=rss 17 15380 W 72.30.161.232 firestats.cc GET /ticket/481 25 15381 W 213.236.203.237 firestats.cc GET /ticket/849?format=rss 26 15377 W 65.55.106.112 firestats.cc GET 30 15100 W 216.158.1.200 firestats.cc GET /ticket/432 30 15353 W 216.158.1.200 firestats.cc GET /ticket/324 38 15351 W 72.30.161.232 firestats.cc GET /ticket/481 55 15348 W 216.158.1.200 firestats.cc GET /ticket/314 56 15350 W 66.249.67.115 firestats.cc GET /ticket/997 58 15349 W 72.30.161.232 firestats.cc GET /ticket/481 59 15329 W 72.30.161.232 firestats.cc GET /wiki/WikiStart?version=152 78 15076 W 121.101.214.54 firestats.cc GET 78 15320 W 121.101.214.54 firestats.cc GET 78 15339 W 121.101.214.54 firestats.cc GET 79 15311 W 72.30.161.232 firestats.cc GET /ticket/112?format=rss 80 15328 W 72.30.161.232 firestats.cc GET /wiki/WikiStart?version=152 100 14987 W 72.30.161.232 firestats.cc GET /ticket/112?format=rss 101 15300 W 72.30.161.232 firestats.cc GET /wiki/WikiStart?version=152 119 15101 W 65.55.106.112 firestats.cc GET 121 15099 W 72.30.161.232 firestats.cc GET /ticket/112?format=rss 122 14970 W 72.30.161.232 firestats.cc GET /wiki/WikiStart?version=152 122 15096 W 216.158.1.200 firestats.cc GET /ticket/1112 142 15095 W 72.30.161.232 firestats.cc GET /ticket/112?format=rss 143 15094 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 148 15084 W 216.158.1.200 firestats.cc GET /ticket/886 149 15068 W 216.158.1.200 firestats.cc GET /ticket/1012 158 15021 W 94.142.63.22 firestats.cc GET / 163 14966 W 72.30.161.232 firestats.cc GET /ticket/481 164 14964 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 175 15060 W 216.158.1.200 firestats.cc GET /ticket/768 183 14965 W 216.158.1.200 firestats.cc GET /ticket/974 184 14929 W 216.158.1.200 firestats.cc GET /wiki/ip2c 184 14952 W 72.30.161.232 firestats.cc GET /ticket/481 185 15022 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 189 15020 W 67.195.114.48 firestats.cc GET /wiki/Standalone?version=13 196 14910 W 65.55.105.205 firestats.cc GET /ticket/106 198 14989 W 220.194.55.47 firestats.cc GET /wiki/Joomla?action=diff&version=12 198 14990 W 79.176.44.160 firestats.cc GET / 205 14985 W 72.30.161.232 firestats.cc GET /ticket/481 206 14972 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 226 14939 W 72.30.161.232 firestats.cc GET /ticket/481 226 14954 W 65.55.105.205 firestats.cc GET /ticket/106 227 14953 W 72.30.161.232 firestats.cc GET /ticket/539 236 14938 W 216.158.1.200 firestats.cc GET /ticket/738 238 14937 W 209.85.238.119 firestats.cc GET /report/7?format=rss&USER=Raptor 239 14863 W 220.194.55.47 firestats.cc GET /wiki/Joomla?action=diff&version=12 248 14892 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 248 14926 W 72.30.161.232 firestats.cc GET /ticket/539 252 14918 W 216.158.1.200 firestats.cc GET /ticket/944 257 14915 W 216.158.1.200 firestats.cc GET /ticket/836 258 14912 W 65.55.105.207 firestats.cc GET /ticket/397?action=diff&version=5 259 14911 W 67.195.114.48 firestats.cc GET /wiki/Standalone?version=13 268 14901 W 216.158.1.200 firestats.cc GET /about 269 14864 W 72.30.161.232 firestats.cc GET /ticket/539 269 14893 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 277 14878 W 91.121.106.59 firestats.cc HEAD / 280 14817 W 67.195.114.48 firestats.cc GET /wiki/Standalone?version=13 280 14891 W 220.194.55.47 firestats.cc GET /wiki/Joomla?action=diff&version=12 288 14820 W 65.55.105.207 firestats.cc GET /ticket/397?action=diff&version=5 289 14879 W 72.30.161.232 firestats.cc GET /ticket/539 290 14877 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 301 14819 W 67.195.114.48 firestats.cc GET /wiki/Standalone?version=13 308 14758 W 91.121.106.59 firestats.cc HEAD / 311 14799 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 311 14848 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 318 14808 W 216.158.1.200 firestats.cc GET /ticket/806 321 14739 W 67.195.114.48 firestats.cc GET /wiki/Standalone?version=13 324 14798 W 220.194.55.47 firestats.cc GET /wiki/Joomla?action=diff&version=12 332 14712 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 332 14787 W 72.30.161.232 firestats.cc GET /ticket/481 338 14760 W 216.158.1.200 firestats.cc GET /ticket/904 340 14759 W 91.121.106.59 firestats.cc HEAD / 343 14815 W 65.55.105.205 firestats.cc GET /ticket/40 353 14806 W 72.30.161.232 firestats.cc GET /ticket/481 353 14807 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 371 14738 W 91.121.106.59 firestats.cc HEAD / 374 14509 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 374 14740 W 72.30.161.232 firestats.cc GET /ticket/481 374 14750 W 65.55.105.205 firestats.cc GET /ticket/40 388 14510 W 67.195.114.48 firestats.cc GET /wiki/Standalone?version=13 394 14716 W 72.30.161.232 firestats.cc GET /ticket/481 395 14715 W 72.30.161.232 firestats.cc GET /ticket/539 398 14714 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 416 14492 W 72.30.161.232 firestats.cc GET /ticket/539 416 14512 W 65.55.207.133 firestats.cc GET /ticket/515 419 14511 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 437 14471 W 220.194.55.47 firestats.cc GET /wiki/Drupal?action=diff&version=9 437 14472 W 72.30.161.232 firestats.cc GET /ticket/539 439 14434 W 79.176.44.160 firestats.cc GET / 440 14463 W 72.30.161.232 firestats.cc GET /ticket/795?version=0 449 14491 W 65.55.207.93 firestats.cc GET /wiki/WikiStart?version=167 458 14424 W 72.30.161.232 firestats.cc GET /ticket/539 462 14448 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 475 14445 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=3 478 14408 W 220.194.55.47 firestats.cc GET /wiki/Drupal?action=diff&version=9 483 14436 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 496 14376 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=3 504 14435 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 507 14375 W 65.55.207.133 firestats.cc GET /ticket/515 508 14423 W 65.55.207.118 firestats.cc GET /ticket/509 517 14407 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=3 518 14406 W 220.194.55.47 firestats.cc GET /wiki/Drupal?action=diff&version=9 524 14366 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 534 14374 W 209.85.238.119 firestats.cc GET /ticket/835?format=rss 534 14381 W 174.129.236.193 firestats.cc GET / 538 14364 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=3 540 14361 W 65.55.207.93 firestats.cc GET /wiki/WikiStart?version=167 541 14382 W 220.194.55.47 firestats.cc GET /wiki/TracModPython 545 14380 W 65.55.105.197 firestats.cc GET /wiki/Features?action=diff&version=13 546 14379 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 559 14351 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=2 559 14365 W 220.194.55.47 firestats.cc GET /wiki/Drupal?action=diff&version=9 565 14330 W 216.158.1.200 firestats.cc GET /export/1984/trunk/firestats/php/page-database.php 567 14353 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 570 14352 W 88.131.106.13 firestats.cc GET /query?status=reopened&milestone=Later 571 14350 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 574 14342 W 216.158.1.200 firestats.cc GET /wiki/InterTrac 575 14306 W 65.55.105.197 firestats.cc GET /wiki/Features?action=diff&version=13 577 14315 W 216.158.1.200 firestats.cc GET /ticket/275 580 14331 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=2 582 14329 W 220.194.55.47 firestats.cc GET /wiki/TracModPython 588 14328 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 592 14292 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 599 14302 W 65.55.207.118 firestats.cc GET /ticket/509 600 14307 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=2 605 14266 W 88.131.106.13 firestats.cc GET /query?status=reopened&milestone=Later 609 14303 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 613 14291 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 621 14232 W 67.195.114.48 firestats.cc GET /wiki/WhoisProviders?version=2 622 14275 W 65.55.105.205 firestats.cc GET /ticket/191 623 14258 W 220.194.55.47 firestats.cc GET /wiki/TracModPython 626 14283 W 216.158.1.200 firestats.cc GET /ticket/255 629 13997 W 216.158.1.200 firestats.cc GET /ticket/147 638 14274 W 88.131.106.13 firestats.cc GET /query?status=reopened&milestone=Later 643 14250 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 653 14252 W 65.55.105.205 firestats.cc GET /ticket/191 655 14251 W 216.158.1.200 firestats.cc GET /ticket/1110 664 14242 W 220.194.55.47 firestats.cc GET /wiki/TracModPython 666 14241 W 65.55.105.197 firestats.cc GET /wiki/ChangeLog?version=67 667 14240 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 670 14207 W 88.131.106.13 firestats.cc GET /query?status=reopened&milestone=Later 679 14216 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 680 14208 W 79.176.44.160 firestats.cc GET / 682 14224 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 688 14223 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 696 13952 W 65.55.105.197 firestats.cc GET /wiki/ChangeLog?version=67 700 13975 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 703 13928 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 708 14193 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 721 14005 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 724 13958 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 741 13927 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 745 13917 W 72.30.161.232 firestats.cc GET /wiki/Mutex?action=history 771 13880 W 216.158.1.200 firestats.cc GET /wiki/trac 778 13949 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 785 13940 W 216.158.1.200 firestats.cc GET /ticket/797 799 9831 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 808 13916 W 67.195.114.48 firestats.cc GET /wiki/CommercialUse?version=10 811 13915 W 65.55.105.197 firestats.cc GET /wiki/Donate?version=4 818 13837 W 220.194.55.45 firestats.cc GET /report/-1?asc=1&sort=report&USER=anonymous 820 13848 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 822 13896 W 65.55.106.155 firestats.cc GET /wiki/MultipleSites 822 13897 W 216.158.1.200 firestats.cc GET /ticket/885 830 13893 W 216.158.1.200 firestats.cc GET /ticket/669 840 13845 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 842 13870 W 65.55.105.197 firestats.cc GET /wiki/Donate?version=4 845 13804 W 216.158.1.200 firestats.cc GET /ticket/875 859 13793 W 220.194.55.45 firestats.cc GET /report/-1?asc=1&sort=report&USER=anonymous 881 13827 W 216.158.1.200 firestats.cc GET /ticket/963 884 13826 W 216.158.1.200 firestats.cc GET /ticket/855 900 13802 W 220.194.55.45 firestats.cc GET /report/-1?asc=1&sort=report&USER=anonymous 900 9834 W 216.158.1.200 firestats.cc GET /ticket/639 907 13792 W 72.30.161.232 firestats.cc GET /ticket/881?format=tab 909 13803 W 216.158.1.200 firestats.cc GET /ticket/845 916 13801 W 216.158.1.200 firestats.cc GET /ticket/629 921 13788 W 79.176.44.160 firestats.cc GET / 921 13791 W 65.55.51.110 firestats.cc GET 924 13780 W 65.55.106.155 firestats.cc GET /wiki/MultipleSites 927 9452 W 216.158.1.200 firestats.cc GET /ticket/943 932 11826 W 216.158.1.200 firestats.cc GET /ticket/835 941 11827 W 220.194.55.45 firestats.cc GET /report/-1?asc=1&sort=report&USER=anonymous 943 9838 W 216.158.1.200 firestats.cc GET /ticket/933 966 9489 W 65.55.105.197 firestats.cc GET /wiki/WikiStart?version=165 997 11825 W 65.55.105.197 firestats.cc GET /wiki/WikiStart?version=165 999 11842 W 216.158.1.200 firestats.cc GET /ticket/805 1073 11844 W 65.55.51.110 firestats.cc GET Mon Sep 28 05:42:06 UTC 2009
