Hi,
I have an application that is freezing at some random times. It
doesn't seem to be obviously related with anything. And restarting
apache seems to fix it.
My first guess was to lower the "maximum-requests" used in WSGI from
500 to 200 to avoid any leak that we might be hitting (open fds, mem,
or something like that), change the LogLevel from warning to info in
apache config and add "inactivity-timeout=600". Just in case, the WSGI
relevant parts in the apache vhost looks like this now:
WSGIDaemonProcess <name> processes=4 maximum-requests=200 threads=1
inactivity-timeout=600
WSGIProcessGroup <some_user>
WSGIScriptAlias / /<path>/apache.wsgi
WSGIApplicationGroup %{GLOBAL}
But with this same config, it happened again today. On the error logs
during that time there several errors like this one:
[Sat Nov 19 02:59:28 2011] [error] [client 127.0.0.1] Script timed
out before returning headers: apache.wsgi
In fact, there are 47 of them. And then 3 like this:
[Sat Nov 19 08:02:25 2011] [info] mod_wsgi (pid=11550): Daemon
process inactivity timer expired, stopping process '<name>'.
And 1 like:
[Sat Nov 19 08:03:06 2011] [info] mod_wsgi (pid=18334): Maximum
requests reached '<name>'.
[Sat Nov 19 08:03:06 2011] [info] mod_wsgi (pid=18334): Shutdown
requested '<name>'.
[Sat Nov 19 08:03:06 2011] [info] mod_wsgi (pid=18334): Stopping
process '<name>'.
[Sat Nov 19 08:03:06 2011] [info] mod_wsgi (pid=18334): Destroying
interpreters.
[Sat Nov 19 08:03:06 2011] [info] mod_wsgi (pid=18334): Cleanup
interpreter ''.
[Sat Nov 19 08:03:07 2011] [info] mod_wsgi (pid=18334): Terminating Python.
[Sat Nov 19 08:03:07 2011] [info] mod_wsgi (pid=18334): Python has shutdown.
After this it all seems to be normal and working again. So it seems
the inactivity time out parameter worked as expected, it wasn't
necessary to restart apache this time.
Also, it seems there is something wrong with the date/hour I didn't
yet look. It didn't happen so much time between the "Script timed out"
and the killing. It seems some messages are logged using one time and
some others using other timezone/something like that. Because I have
this in the logs, for example:
[Sat Nov 19 08:04:31 2011] [info] mod_wsgi (pid=13853): Aborting
process '<name>'.
[Sat Nov 19 03:04:31 2011] [error] [client 127.0.0.1] Premature end of
script headers: apache.wsgi
[Sat Nov 19 08:04:31 2011] [info] mod_wsgi (pid=18356): Python has shutdown.
[Sat Nov 19 03:04:32 2011] [info] mod_wsgi (pid=18404): Attach interpreter ''.
But the time is actually bad configured on that server (is not on our
timezone and I will fix it as soon I can confirm the application
wouldn't have any problems with that), so perhaps it is reported in
some place as one timezone and some other in some other config file.
This application uses a Mysql (percona, a fork of mysql actually)
hosted on some other machine. Looking the munin and new-relic stats I
don't see anything weird. Although in muning, with the mysql plugin, I
see a huge increase for some minutes of "binlog cache usage". That's
the only "anomaly" I see.
Anyone has any idea/tip on how can I further debug this ?
I've seen there are commits in WSGI (not included in any release yet)
that add a blocked timeout and blocker request parameters and an other
one to get some information of the processes/threads (which
information ? a call-trace perhaps ? :-)). If I understand correctly
this could be *very* helpful to debug this problem. Also, if we could
have a trace where the blocked process where hanging it will be *very
very* useful :-D
Is there some other configuration option that could help ? Do you know
when will be a new release that includes the "blocked request"
parameters ?
Oh, btw, I'm using libapache2-mod-wsg 3.3-2ubuntu2
Also, if you need more information, please let me know.
Thanks a lot,
Rodrigo
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en.
