This is great that a new locking implementation will be soon :) Many thanks. But looking at behavior it is seems more like a harakiri bug to me. Locks works as expected in usual. (if you kill a process owning lock others deadlock). Harakiri system seems to be based on fact that usr2 will iterrupt blocking call and after C level usr2 callback execution of request handler continues. (which is a bug) As for the master-governed locking system: anything like that will flow like this: * worker says to master to make lock[i]. acquire() (make a syscall) * a master process should be waked up on any available core.. read worker request to make locking and make it (at least 2 syscall) * after locking is made send message to worker (make a syscall) saying that locking is made * a worker should be waked up on any available core and continue And the same scheme with .release().
It is seems like overkill to make process switch to make lock. acquire() (not to mention that it is a lot more syscalls than 1 which is required to make mutex acquire). Compared to this pthread mutex + robust attribute seems like a decent solution. Best regards. Many thanks for cool and fast code :) 2012/2/1 Roberto De Ioris <[email protected]>: > >> I am sorry second call: kill(uwsgi.workers[i].pid, SIGKILL); obviously. > > i have made same test, it seems only the flock() implementation is immune > 100% of the times to the problem. But flock is slow as hell (it is the > default only on OpenBSD, NetBSD and older linux kernel). > > I think it is time to throw away the current implementation and use some > master-governed locking system. This will bypass platform-dependent > problems and will allows new technics (like forcing unlocking and similar > paradigms) > > As this has beaten me in the head (i have a bunch of aplications relaying > on that) i will came up with an implementation really soon (i hope today) > and i will backport it to 1.0 for sure. > > Thanks for your report > > > > >> >> 2012/2/1 Evgeny Turnaev <[email protected]>: >>> Linux nostromo 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC >>> 2011 x86_64 x86_64 x86_64 GNU/Linux >>> uWSGI 1.0-dev (uwsgi-1.0-rc7) >>> Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53) >>> >>> I think master sends USR2, worker interrupts first sleep in def.. then >>> continues to seconds sleep and after 1 sec of sleep master sends >>> kill(uwsgi.workers[i].pid, SIGUSR2); which will lead in killing worker >>> holding lock and thus no other request will proceed. >>> >>> 2012/2/1 Roberto De Ioris <[email protected]>: >>>> >>>>> I am sorry but uwsgi.lock() can deadlock. Here is an example: >>>>> >>>>> #!/usr/local/bin/python2.7 >>>>> >>>>> import uwsgi >>>>> import os >>>>> import time >>>>> import random >>>>> >>>>> >>>>> def do_lock_thing(): >>>>>   print "pid: %s going to lock" % os.getpid() >>>>>   uwsgi.lock() >>>>>   print "pid: %s locked." % os.getpid() >>>>>   sleep_time = 8 >>>>>   print "pid: %s going to sleep: %0.2f" % (os.getpid(), >>>>> sleep_time) >>>>>   time.sleep(sleep_time) >>>>>   print "pid: %s going again to sleep: %0.2f" % (os.getpid(), >>>>> sleep_time) >>>>>   time.sleep(sleep_time) >>>>>   print "pid: %s going to UNlock" % os.getpid() >>>>>   uwsgi.unlock() >>>>> >>>>> def application(environ, start_response): >>>>>   headers = [ >>>>>     ('Content-Type', 'text/html'), >>>>>   ] >>>>>   write = start_response('200 OK', headers) >>>>>   print "pid: %s request started" % os.getpid() >>>>>   time.sleep(5) >>>>>   for x in range(2): >>>>>     do_lock_thing() >>>>>     yield 'Hello World' >>>>>   print "pid: %s end of application.." % os.getpid() >>>>> >>>>> run as >>>>> sudo uwsgi -s /var/run/uwsgi_hypernova.sock -M -p 2 --harakiri 10 >>>>> --harakiri-verbose --pythonpath . -w uwsgi_test4_lock_in_def -C 777 >>>>> >>>>> The first process locks lock and then after it get kharakiri killed >>>>> any other request will be blocked in uwsgi.lock() forever. >>>>> >>>>> pid: 7969 request started >>>>> pid: 7970 request started >>>>> pid: 7969 going to lock >>>>> pid: 7969 locked. >>>>> pid: 7969 going to sleep: 8.00 >>>>> pid: 7970 going to lock >>>>> *** HARAKIRI ON WORKER 1 (pid: 7969) *** >>>>> HARAKIRI: -- syscall> 23 0x0 0x0 0x0 0x0 0x7fff5b298a00 0x1ffff >>>>> 0x7fff5b298770 0x7f3ed5d08123 >>>>> HARAKIRI: -- wchan> poll_schedule_timeout >>>>> HARAKIRI: --- uWSGI worker 1 (pid: 7969) WAS managing request >>>>> /serp?query=%D0%BF%D1%83%D1%82%D0%B8%D0%BD since Wed Feb  1 13:31:45 >>>>> 2012 --- >>>>> pid: 7969 going again to sleep: 8.00 >>>>> *** HARAKIRI ON WORKER 2 (pid: 7970) *** >>>>> HARAKIRI: -- syscall> 202 0x7f3ed799c0a0 0x0 0x2 0x0 0x7f3ed799c0a0 >>>>> 0x1f22 0x7fff5b298a00 0x7f3ed756f7c4 >>>>> HARAKIRI: -- wchan> futex_wait_queue_me >>>>> HARAKIRI: --- uWSGI worker 2 (pid: 7970) WAS managing request >>>>> /serp?query=%D0%BF%D1%83%D1%82%D0%B8%D0%BD since Wed Feb  1 13:31:46 >>>>> 2012 --- >>>>> DAMN ! worker 1 (pid: 7969) died :( trying respawn ... >>>>> Respawned uWSGI worker 1 (new pid: 7971) >>>>> DAMN ! worker 2 (pid: 7970) died :( trying respawn ... >>>>> Respawned uWSGI worker 2 (new pid: 7972) >>>>> pid: 7971 request started >>>>> pid: 7971 going to lock >>>>> *** HARAKIRI ON WORKER 1 (pid: 7971) *** >>>>> HARAKIRI: -- syscall> 202 0x7f3ed799c0a0 0x0 0x2 0x0 0x7f3ed799c0a0 >>>>> 0x1f23 0x7fff5b298a00 0x7f3ed756f7c4 >>>>> HARAKIRI: -- wchan> futex_wait_queue_me >>>>> HARAKIRI: --- uWSGI worker 1 (pid: 7971) WAS managing request >>>>> /serp?query=%D0%BF%D1%83%D1%82%D0%B8%D0%BD since Wed Feb  1 13:32:08 >>>>> 2012 --- >>>>> pid: 7972 request started >>>>> DAMN ! worker 1 (pid: 7971) died :( trying respawn ... >>>>> Respawned uWSGI worker 1 (new pid: 7974) >>>>> pid: 7972 going to lock >>>>> ^CSIGINT/SIGQUIT received...killing workers... >>>>> goodbye to uWSGI. >>>>> >>>>> >>>>> >>>> >>>> This is a bug in uwsgi.lock()/unlock(), which kernel version ? >>>> The usage of futex syscall means you are correctly running on >>>> pthread_mutex, so it is a uWSGI problem for sure. >>>> >>>> I cannot reproduce it on linux 3.0 >>>> >>>> >>>> -- >>>> Roberto De Ioris >>>> http://unbit.it >>>> _______________________________________________ >>>> uWSGI mailing list >>>> [email protected] >>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>> >>> >>> >>> -- >>> -------------------------------------------- >>> Турнаев Евгений Викторович >>> +7 906 875 09 43 >>> -------------------------------------------- >> >> >> >> -- >> -------------------------------------------- >> Турнаев Евгений Викторович >> +7 906 875 09 43 >> -------------------------------------------- >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> > > > -- > Roberto De Ioris > http://unbit.it > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi -- -------------------------------------------- Турнаев Евгений Викторович +7 906 875 09 43 -------------------------------------------- _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
