I found the cause behind why the workers were being killed. A few
weeks ago I added the following function in the uwsgi_utils.py file
(see config from original post):
@timer(900)
def update_directory_visible_cache(num):
import cPickle as pickle
from django.core.cache import cache
from businesses.models import Business
dir_visible_list = []
for business in Business.objects.filter(biz_status='ACTIVE'):
if business.directory_visible:
dir_visible_list.append(business.pk)
cache.set('dir_visible_list', pickle.dumps(dir_visible_list), 1800)
After removing this function, everything has been working perfectly
over the last 2 days and no workers have been killed (that I can see,
at least) in the logs. After reviewing the old logs more thoroughly,
I noticed several lines like this:
registered signal 66
you can register max 64 timers !!!
registered signal 67
you can register max 64 timers !!!
registered signal 68
you can register max 64 timers !!!
etc etc..
I currently have no way to confirm that the timer jobs were actually
being executed, I guess I could re-implement this on my dev site and
print debug info from the timing jobs if you guys need more info.
I hope this helps!
Ryan-
On Sat, Oct 15, 2011 at 12:21 PM, Roberto De Ioris <[email protected]> wrote:
>
>> On Fri, Oct 14, 2011 at 10:35 PM, Roberto De Ioris <[email protected]>
>> wrote:
>>>
>>>> Unfortunately the issue persists, albeit at a lesser scale. Â Spooler
>>>> jobs were successfully created this morning and the workers don't
>>>> appear to be dying off quite as often. Â The config is the same as I
>>>> had posted before minus the max-requests and optimize flags. Â Here are
>>>> two excerpts from the logs:
>>>>
>>>>
>>>
>>> Ok, this confirm an issue with max-requests and the spooler (propagating
>>> to workers).
>>>
>>> Your latest errors seems triggered by the signal handling subsystem.
>>> I do not know if you need to spawn threads in your app, but if you do
>>> not
>>> need it, please disable --enable-threads to see if the signal-handler
>>> crash is triggered by a broken GIL lock.
>>
>> Unfortunately I am currently using the enable-threads option to send
>> out emails. It looks something like this:
>>
>> queues = {}
>>
>> class queueconsumer(object):
>> def __init__(self, name, num=1, **kwargs):
>> self.name = name
>> self.num = num
>> self.queue = Queue.Queue()
>> self.threads = []
>> self.func = None
>> queues[self.name] = self
>>
>> @staticmethod
>> def consumer(self):
>> while True:
>> req = self.queue.get()
>> self.func(req)
>> self.queue.task_done()
>>
>> def __call__(self, f):
>> self.func = f
>> for i in range(self.num):
>> t = Thread(target=self.consumer, args=(self,))
>> self.threads.append(t)
>> t.daemon = True
>> t.start()
>>
>>
>> @queueconsumer('email', 2)
>> def send_email_task(args):
>> from django.core.mail import EmailMultiAlternatives
>> from django.utils import simplejson
>>
>> body = args['body']
>> email_details = simplejson.loads(body)
>> email = EmailMultiAlternatives(**email_details)
>> email.send(fail_silently=False)
>>
>> It's basically taken from the examples you guys posted a while ago for
>> multi-threaded spool jobs, but I don't create a spool job in this
>> situation, I just queue the email to be sent into the dictionary and
>> let the threads take care of it. I can re-work the code if you guys
>> would like me to see if this is causing the issue or not.
>>
>> Also, I spoke too soon when I said that spooler jobs were being
>> created again, after being left running long enough, the spooler
>> system shut down and no longer created spool jobs or executed spool
>> jobs that had been left in it's spool.
>>
>> Thanks for all the help on this,
>> Ryan-
>
>
> The problem with python threads is that you can lose exceptions, and the
> result is in your spooler process silently dying. If the master process
> see a process respawning too fast, it slowdown to avoid fork bombing. So i
> assume this is your current status.
>
> For some reason one of the thread in your spooler is crashing the whole
> subsystem, but you do not get the exception.
>
> If you can re-work this part of your app to be sure the problem is with
> threads+spooleer it would be useful (check for mules in 1.0 branch). In
> the mean time i will check if we can find a workaround to get threads
> exception asynchronously.
>
> --
> Roberto De Ioris
> http://unbit.it
> _______________________________________________
> uWSGI mailing list
> [email protected]
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
