2009/2/16 gert <[email protected]>:
>
> On Feb 16, 12:20 am, Graham Dumpleton <[email protected]>
> wrote:
>> 2009/2/16 gert <[email protected]>:
>>
>> > On Feb 15, 11:57 pm, gert <[email protected]> wrote:
>> >> On Feb 15, 11:34 pm, Graham Dumpleton <[email protected]>
>> >> wrote:
>>
>> >> > 2009/2/16 gert <[email protected]>:
>>
>> >> > > On Feb 15, 11:04 pm, Graham Dumpleton <[email protected]>
>> >> > > wrote:
>> >> > >> 2009/2/16 gert <[email protected]>:
>>
>> >> > >> > [Sun Feb 15 22:39:22 2009] [info] mod_wsgi (pid=9850): Destroy
>> >> > >> > interpreter '127.0.0.1|/appwsgi/www/order/order.wsgi'.
>> >> > >> > [Sun Feb 15 22:39:22 2009] [info] mod_wsgi (pid=9850): Destroy
>> >> > >> > interpreter '127.0.0.1|/appwsgi/www/query/query.wsgi'.
>> >> > >> > [Sun Feb 15 22:39:22 2009] [info] mod_wsgi (pid=9850): Cleanup
>> >> > >> > interpreter ''.
>> >> > >> > [Sun Feb 15 22:39:22 2009] [info] mod_wsgi (pid=9850): Terminating
>> >> > >> > Python.
>> >> > >> > [Sun Feb 15 22:39:23 2009] [notice] child pid 9850 exit signal
>> >> > >> > Segmentation fault (11)
>>
>> >> > >> > some reloading bug ?
>> >> > >> > very hard to reproduce
>> >> > >> > [Sun Feb 15 23:02:57 2009] [info] mod_wsgi (pid=9789): Destroy
>> >> > >> > interpreter '127.0.0.1|/appwsgi/www/order/order.wsgi'.
>> >> > >> > [Sun Feb 15 23:02:57 2009] [info] mod_wsgi (pid=9789): Cleanup
>> >> > >> > interpreter ''.
>> >> > >> > [Sun Feb 15 23:02:57 2009] [info] mod_wsgi (pid=9789): Terminating
>> >> > >> > Python.
>> >> > >> > [Sun Feb 15 23:02:58 2009] [notice] child pid 9789 exit signal
>> >> > >> > Segmentation fault (11)
>>
>> >> > >> > Ok it happens when you edit a live wsgi file i think
>>
>> >> > >> What do you mean by reloading bug?
>>
>> >> > >> Is this from touching the WSGI script file, or from executing
>> >> > >> 'apachectl'? If the latter, what argument.
>>
>> >> > >> If from touching WSGI script, thus using daemon mode, then attach gdb
>> >> > >> to process as per debugging instructions in wiki and try and capture
>> >> > >> stack trace.
>>
>> >> > >> BTW, since you are playing with multiple versions of Python and of
>> >> > >> mod_wsgi, state which versions.
>>
>> >> > > not touching apachectl
>> >> > > embedded mode (http://code.google.com/p/appwsgi/source/browse/trunk/
>> >> > > httpd.conf)
>> >> > > mod r1200 py3 apache 2.2.11
>>
>> >> > > doing the wiki stuff, the thing is, it does not crash on my command
>> >> > > only when there are a BUNCH of processes who needs to get updated.
>>
>> >> > You must be using apachectl. Alternatively, you must have
>> >> > MaxRequestsPerChild set for MPM such that child server processes
>> >> > restarted on periodic basis.
>>
>> >> > Merely editing the WSGI script file will not trigger a child server
>> >> > process shutdown and so you wouldn't see an attempt to terminate
>> >> > Python.
>>
>> >> > So, do you have MaxRequestsPerChild set then?
>>
>> >> No no no apachectl, everything default there is no MaxRequestsPerChild
>> >> defined in my httpd, it is exactly the same as the one from the url
>> >> above
>>
>> >> PS is it usefull to not start it with -X because it will not crash
>> >> when using -X i think
>>
>> >> r...@r12276:/usr/httpd/bin# gdb ./httpd
>> >> GNU gdb 6.8-debian
>> >> Copyright (C) 2008 Free Software Foundation, Inc.
>> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
>> >> gpl.html>
>> >> This is free software: you are free to change and redistribute it.
>> >> There is NO WARRANTY, to the extent permitted by law. Type "show
>> >> copying"
>> >> and "show warranty" for details.
>> >> This GDB was configured as "i486-linux-gnu"...
>> >> (gdb) run -X
>> >> Starting program: /usr/httpd/bin/httpd -X
>> >> [Thread debugging using libthread_db enabled]
>> >> [New Thread 0xb7e196b0 (LWP 14629)]
>> >> [New Thread 0xb7826b90 (LWP 14834)]
>> >> [Thread 0xb7826b90 (LWP 14834) exited]
>> >> [New Thread 0xb7025b90 (LWP 14835)]
>> >> [Thread 0xb7025b90 (LWP 14835) exited]
>>
>> > [Mon Feb 16 00:01:40 2009] [info] [client 217.136.56.216] mod_wsgi
>> > (pid=14629, process='', application='127.0.0.1|/appwsgi/www/order/
>> > order.wsgi'): Reloading WSGI script '/usr/httpd/www/appwsgi/www/order/
>> > order.wsgi'., referer:http://91.121.53.159/appwsgi/www/order/order.htm
>>
>> > The reloading happens nicely now with the -X but when there are a
>> > bunch of processes getting harvest in the middle of a reloading you
>> > will have a segfault party :) Now how to reproduce that i have no idea.
>>
>> What has confused things is your use of term 'reloading' as it made me
>> forget about one other thing that can be happening. That is that you
>> have been benchmarking and the load has caused Apache to create
>> additional child server processes. When the benchmarking is done,
>> Apache will start to shutdown the then idle child server processes,
>> triggering the destruction of interpreters, Python etc.
>>
>> So, have you then been doing something that would cause Apache to
>> create additional child server processes beyond the minimum it
>> creates?
>
> My site spams allot of small ajax request :-)
>
>> BTW, do you create any background threads from your application?
>
> nope but i have this allot
>
> [Mon Feb 16 00:24:20 2009] [info] mod_wsgi (pid=17142): Cleanup
> interpreter ''.
> [Mon Feb 16 00:24:20 2009] [info] mod_wsgi (pid=17142): Terminating
> Python.
> [Mon Feb 16 00:24:20 2009] [info] mod_wsgi (pid=17142): Python has
> shutdown.
> [Mon Feb 16 00:24:21 2009] [info] mod_wsgi (pid=17143): Cleanup
> interpreter ''.
> [Mon Feb 16 00:24:21 2009] [info] mod_wsgi (pid=17143): Terminating
> Python.
> [Mon Feb 16 00:24:21 2009] [info] mod_wsgi (pid=17143): Python has
> shutdown.
>
> so i am pretty sure when i edit a wsgi file while apache is doing the
> above i will have a segfault
Editing the WSGI file while this happening would have no affect, as
once child server process starts to shutdown it isn't handing
requests, so doesn't care if your file changes.
Now, if you are seeing lots of these messages then you are possibly a
classic example of those people who I am having a bit of a winge about
in blog post I am writing at the moment. That is, using embedded mode,
especially with prefork, but not bothering to adjust the MPM settings
to something more appropriate for Python web applications.
Are you still using prefork MPM? What are the MPM settings you are
using? The defaults for prefork are:
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
What do you have? Or as usual have you completely butchered your
Apache configuration file so it more or less has nothing in it,
including no settings for those values?
Back to the crash. I recently added the additional message:
[Mon Feb 16 00:24:21 2009] [info] mod_wsgi (pid=17143): Python has shutdown.
Are you seeing this ever time for corresponding message above it of:
[Mon Feb 16 00:24:21 2009] [info] mod_wsgi (pid=17143): Terminating Python.
If you aren't and you see the segmentation fault in some or all cases,
then the crash is occuring when calling Py_Finalize() on Python
interpreter.
I also recently changed how thread state is acquired for calling
Py_Finalize(). It used to create a new thread state just for that
operation, but now it uses simplified GIL state API as it by rights
should do. Before we look though at reverting that bit of code, answer
the above questions about your configuration and the log messages.
Graham
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en
-~----------~----~----~----~------~----~------~--~---
