On Wed, Oct 19, 2011 at 12:15 PM, Ryan Showalter <[email protected]> wrote: > On Wed, Oct 19, 2011 at 11:47 AM, Roberto De Ioris <[email protected]> wrote: >> >>> On Wed, Oct 19, 2011 at 11:22 AM, Roberto De Ioris <[email protected]> >>> wrote: >>>> >>>>> I found the cause behind why the workers were being killed. Â A few >>>>> weeks ago I added the following function in the uwsgi_utils.py file >>>>> (see config from original post): >>>>> >>>>> @timer(900) >>>>> def update_directory_visible_cache(num): >>>>> Â Â import cPickle as pickle >>>>> >>>>> Â Â from django.core.cache import cache >>>>> >>>>> Â Â from businesses.models import Business >>>>> >>>>> Â Â dir_visible_list = [] >>>>> Â Â for business in Business.objects.filter(biz_status='ACTIVE'): >>>>> Â Â Â Â if business.directory_visible: >>>>> Â Â Â Â Â Â dir_visible_list.append(business.pk) >>>>> >>>>> Â Â cache.set('dir_visible_list', pickle.dumps(dir_visible_list), >>>>> 1800) >>>>> >>>>> >>>>> After removing this function, everything has been working perfectly >>>>> over the last 2 days and no workers have been killed (that I can see, >>>>> at least) in the logs. Â After reviewing the old logs more thoroughly, >>>>> I noticed several lines like this: >>>>> >>>>> registered signal 66 >>>>> you can register max 64 timers !!! >>>>> >>>>> registered signal 67 >>>>> you can register max 64 timers !!! >>>>> >>>>> registered signal 68 >>>>> you can register max 64 timers !!! >>>>> >>>>> etc etc.. >>>> >>>> >>>> This condition should raise a python exception: >>>> >>>> ValueError("unable to add timer") >>>> >>>> can you check if it is raised ? (you should see the message in the logs >>>> soon after the "you can register max 64 timers !!!" error) >>> >>> I do in fact see the following in my stdout logs: >>> >>> ... blah blah... >>> >>> unable to add timer >>> ... blah blah blah... >>> >>> But since the "you can register max 64 timers !!!" is in my stderr >>> logs and "unable to add timer" is in the stdout logs, it's impossible >>> for me to correlate the two since neither of the messages in either of >>> the logs have any sort of timestamps. >>> >> >> Signals are registered ASAP so if your module define function at address >> 0xAAAAAAAA uWSGI will put this address in the signal table. >> >> But at a point in your module code, an exception is raised and all of the >> module memory is freed and 0xAAAAAAAA is no more valid. > > Where exactly does it become an issue when an exception is raised? > Raising exceptions is an essential part of my @spool functions so that > they keep retrying (SPOOL_RETRY) until the function returns > successfully and thus SPOOL_OK is sent to uwsgi
Ah, I just re-read the thread and noticed that you said that the rogue exception may be escaping from the spool thread somewhere. If this is the case, why do you think the spooler exception is getting through (and then possibly crashing the subsystem) only when timers are being registered? As far as I can tell, the timer is not raising any sort of exception when they're actually being executed. > >> >> The signal subsystem has no way to know this, so it will simply call an >> undefined memory area and the worker crashes. >> >> Honestly i see no way to fix this behaviour (the signal subsystem should >> rollback all of the registered signals, and this is unpractical). By the >> way the 64 timer limit is only "theoretical" this limit could be raised in >> the code without problems (or made configurable). In the mean time you can >> use @rbtimer instead of @timer for timers > 64 (you can mix rbtimers and >> timers without problems) >> >> >> >> -- >> Roberto De Ioris >> http://unbit.it >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> > _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
