Hi Folks, I was attempting to upload new git packages this morning and was getting 503's from the csw-upload-pkg script. Visiting the url http://buildfarm.opencsw.org/pkgdb/ yielded the same.
Digging into the backend, it looks like there is a wsgi issue that has resulting in apache not properly respawing the backend processes: [Sat May 31 04:24:26 2014] [info] [client 213.178.77.176] mod_wsgi (pid=12178, process='buildfarm.opencs w.org', application='buildfarm.opencsw.org|/releases'): Loading WSGI script '/home/web/bin/gar/lib/web/r eleases_web.py'. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Maximum requests reached 'buildfarm.opencsw.org' . [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Shutdown requested 'buildfarm.opencsw.org'. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Stopping process 'buildfarm.opencsw.org'. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Destroying interpreters. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Destroy interpreter 'buildfarm.opencsw.org|/rele ases'. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Cleanup interpreter ''. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Terminating Python. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Python has shutdown. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12207): Attach interpreter ''. [Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12207): Create interpreter 'buildfarm.opencsw.org|/relea ses'. [Sat May 31 04:24:52 2014] [info] [client 213.178.77.176] mod_wsgi (pid=12207, process='buildfarm.opencs w.org', application='buildfarm.opencsw.org|/releases'): Loading WSGI script '/home/web/bin/gar/lib/web/r eleases_web.py'. [Sat May 31 04:25:00 2014] [info] mod_wsgi (pid=12207): Create interpreter 'buildfarm.opencsw.org|/pkgdb '. [Sat May 31 04:25:00 2014] [info] [client 213.178.77.178] mod_wsgi (pid=12207, process='buildfarm.opencs w.org', application='buildfarm.opencsw.org|/pkgdb'): Loading WSGI script '/home/web/bin/gar/lib/web/pkgd b_web.py'. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Maximum requests reached 'buildfarm.opencsw.org' . [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Shutdown requested 'buildfarm.opencsw.org'. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Stopping process 'buildfarm.opencsw.org'. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Destroying interpreters. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Destroy interpreter 'buildfarm.opencsw.org|/rele ases'. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Destroy interpreter 'buildfarm.opencsw.org|/pkgd b'. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Cleanup interpreter ''. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Terminating Python. [Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Python has shutdown. [..snip lines from other stuff...] [Sat May 31 04:30:21 2014] [error] [client 213.178.77.176] Script timed out before returning headers: re leases_web.py [Sat May 31 04:35:00 2014] [error] [client 213.178.77.178] Script timed out before returning headers: pk gdb_web.py [Sat May 31 04:35:24 2014] [error] [client 213.178.77.176] Script timed out before returning headers: pk gdb_web.py [Sat May 31 04:37:02 2014] [error] [client 213.178.77.176] Script timed out before returning headers: pk gdb_web.py [Sat May 31 04:40:00 2014] [error] [client 213.178.77.178] Script timed out before returning headers: pk gdb_web.py This is now resulting in errors logged for each connection attempt: [Sat May 31 08:00:00 2014] [error] [client 213.178.77.178] (146)Connection refused: mod_wsgi (pid=11958): Connection attempt #1 to WSGI daemon process 'buildfarm.opencsw.org' on '/var/run/wsgi.20108.0.1.sock' failed, sleeping before retrying again. [Sat May 31 08:00:00 2014] [error] [client 213.178.77.178] (146)Connection refused: mod_wsgi (pid=11958): Connection attempt #2 to WSGI daemon process 'buildfarm.opencsw.org' on '/var/run/wsgi.20108.0.1.sock' failed, sleeping before retrying again. [Sat May 31 08:00:00 2014] [error] [client 213.178.77.178] Script timed out before returning headers: pkgdb_web.py I just kicked apache and that seems to have brought things back to life. I'm not currently planning to dig any further for a root cause here, but am noting the event in case it becomes a pattern in the future - we'll have more data to work with. Thanks -Ben
