On Wed, Oct 19, 2011 at 11:47 AM, Roberto De Ioris <[email protected]> wrote: > >> On Wed, Oct 19, 2011 at 11:22 AM, Roberto De Ioris <[email protected]> >> wrote: >>> >>>> I found the cause behind why the workers were being killed. Â A few >>>> weeks ago I added the following function in the uwsgi_utils.py file >>>> (see config from original post): >>>> >>>> @timer(900) >>>> def update_directory_visible_cache(num): >>>> Â Â import cPickle as pickle >>>> >>>> Â Â from django.core.cache import cache >>>> >>>> Â Â from businesses.models import Business >>>> >>>> Â Â dir_visible_list = [] >>>> Â Â for business in Business.objects.filter(biz_status='ACTIVE'): >>>> Â Â Â Â if business.directory_visible: >>>> Â Â Â Â Â Â dir_visible_list.append(business.pk) >>>> >>>> Â Â cache.set('dir_visible_list', pickle.dumps(dir_visible_list), >>>> 1800) >>>> >>>> >>>> After removing this function, everything has been working perfectly >>>> over the last 2 days and no workers have been killed (that I can see, >>>> at least) in the logs. Â After reviewing the old logs more thoroughly, >>>> I noticed several lines like this: >>>> >>>> registered signal 66 >>>> you can register max 64 timers !!! >>>> >>>> registered signal 67 >>>> you can register max 64 timers !!! >>>> >>>> registered signal 68 >>>> you can register max 64 timers !!! >>>> >>>> etc etc.. >>> >>> >>> This condition should raise a python exception: >>> >>> ValueError("unable to add timer") >>> >>> can you check if it is raised ? (you should see the message in the logs >>> soon after the "you can register max 64 timers !!!" error) >> >> I do in fact see the following in my stdout logs: >> >> ... blah blah... >> >> unable to add timer >> ... blah blah blah... >> >> But since the "you can register max 64 timers !!!" is in my stderr >> logs and "unable to add timer" is in the stdout logs, it's impossible >> for me to correlate the two since neither of the messages in either of >> the logs have any sort of timestamps. >> > > Signals are registered ASAP so if your module define function at address > 0xAAAAAAAA uWSGI will put this address in the signal table. > > But at a point in your module code, an exception is raised and all of the > module memory is freed and 0xAAAAAAAA is no more valid.
Where exactly does it become an issue when an exception is raised? Raising exceptions is an essential part of my @spool functions so that they keep retrying (SPOOL_RETRY) until the function returns successfully and thus SPOOL_OK is sent to uwsgi > > The signal subsystem has no way to know this, so it will simply call an > undefined memory area and the worker crashes. > > Honestly i see no way to fix this behaviour (the signal subsystem should > rollback all of the registered signals, and this is unpractical). By the > way the 64 timer limit is only "theoretical" this limit could be raised in > the code without problems (or made configurable). In the mean time you can > use @rbtimer instead of @timer for timers > 64 (you can mix rbtimers and > timers without problems) > > > > -- > Roberto De Ioris > http://unbit.it > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
