I am sorry but uwsgi.lock() can deadlock. Here is an example:
#!/usr/local/bin/python2.7
import uwsgi
import os
import time
import random
def do_lock_thing():
print "pid: %s going to lock" % os.getpid()
uwsgi.lock()
print "pid: %s locked." % os.getpid()
sleep_time = 8
print "pid: %s going to sleep: %0.2f" % (os.getpid(), sleep_time)
time.sleep(sleep_time)
print "pid: %s going again to sleep: %0.2f" % (os.getpid(), sleep_time)
time.sleep(sleep_time)
print "pid: %s going to UNlock" % os.getpid()
uwsgi.unlock()
def application(environ, start_response):
headers = [
('Content-Type', 'text/html'),
]
write = start_response('200 OK', headers)
print "pid: %s request started" % os.getpid()
time.sleep(5)
for x in range(2):
do_lock_thing()
yield 'Hello World'
print "pid: %s end of application.." % os.getpid()
run as
sudo uwsgi -s /var/run/uwsgi_hypernova.sock -M -p 2 --harakiri 10
--harakiri-verbose --pythonpath . -w uwsgi_test4_lock_in_def -C 777
The first process locks lock and then after it get kharakiri killed
any other request will be blocked in uwsgi.lock() forever.
pid: 7969 request started
pid: 7970 request started
pid: 7969 going to lock
pid: 7969 locked.
pid: 7969 going to sleep: 8.00
pid: 7970 going to lock
*** HARAKIRI ON WORKER 1 (pid: 7969) ***
HARAKIRI: -- syscall> 23 0x0 0x0 0x0 0x0 0x7fff5b298a00 0x1ffff
0x7fff5b298770 0x7f3ed5d08123
HARAKIRI: -- wchan> poll_schedule_timeout
HARAKIRI: --- uWSGI worker 1 (pid: 7969) WAS managing request
/serp?query=%D0%BF%D1%83%D1%82%D0%B8%D0%BD since Wed Feb 1 13:31:45
2012 ---
pid: 7969 going again to sleep: 8.00
*** HARAKIRI ON WORKER 2 (pid: 7970) ***
HARAKIRI: -- syscall> 202 0x7f3ed799c0a0 0x0 0x2 0x0 0x7f3ed799c0a0
0x1f22 0x7fff5b298a00 0x7f3ed756f7c4
HARAKIRI: -- wchan> futex_wait_queue_me
HARAKIRI: --- uWSGI worker 2 (pid: 7970) WAS managing request
/serp?query=%D0%BF%D1%83%D1%82%D0%B8%D0%BD since Wed Feb 1 13:31:46
2012 ---
DAMN ! worker 1 (pid: 7969) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 7971)
DAMN ! worker 2 (pid: 7970) died :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 7972)
pid: 7971 request started
pid: 7971 going to lock
*** HARAKIRI ON WORKER 1 (pid: 7971) ***
HARAKIRI: -- syscall> 202 0x7f3ed799c0a0 0x0 0x2 0x0 0x7f3ed799c0a0
0x1f23 0x7fff5b298a00 0x7f3ed756f7c4
HARAKIRI: -- wchan> futex_wait_queue_me
HARAKIRI: --- uWSGI worker 1 (pid: 7971) WAS managing request
/serp?query=%D0%BF%D1%83%D1%82%D0%B8%D0%BD since Wed Feb 1 13:32:08
2012 ---
pid: 7972 request started
DAMN ! worker 1 (pid: 7971) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 7974)
pid: 7972 going to lock
^CSIGINT/SIGQUIT received...killing workers...
goodbye to uWSGI.
2012/2/1 Roberto De Ioris <[email protected]>:
>
>> Hello Roberto.
>> My application use some shared memory black magic. (it is a legacy
>> code)
>> For sync i am using python multiprocessing.Semaphore.
>>
>> And i am a bit concerned about possible deadlock if harakiri going
>> to happen when one worker holds a lock.
>> I saw a uwsgi.after_request (did not tried it yet) in a mail list
>> recently so one can use it to do sem.release() in there.
>> Šukasz Wróblewski is asking for after_request to get a parameter
>> on how request was finished: (success, error, harakiri) and i agree
>> with him that would be really cool feature for cleanup/stats system of
>> applications.
>>
>> Thoughts about harakiri and deadlock:
>> With current implementation of harakiri it is seems master is
>> sending USR2 to worker that is stucked. Suppose worker is stucked in
>> time.sleep() then USR2 going to interrupt sleep syscall and call USR2
>> handler (that will print what a worker is doing into uwsgi log). The
>> bad thing is that after usr2 signal handler is done execution of
>> request handler continues and there can be another blocking call
>> (sleep for example).. and then a master will send sigkill after 1 sec.
>> The bad thing about continuing request handler execution - that causes
>> hard (evil) killing of worker if it continues to process request.
>> Anyway with continuing request handler or not after harakiri on
>> worker application may be deadlocked in both cases when it uses
>> uwsgi.lock and multiprocessing.Lock. after_request hook seems to help
>> a bit (as it will interrupt blocking call) but not much - because if
>> then in request handler more work to do - it will be killed with
>> sigkill.
>>
>> a possible solution (not really cool solution) is to handle harakiri
>> so:
>> master worker
>> send usr2
>> intercepts usr2 & prints current
>> state to log (as now)
>> calls after_request (without
>> continuing request_handeler)
>> if after_request also blocks
>> sends usr3
>> intercepts usr3
>> for lock in
>> uwsgi.lock[max_lock]: lock.release(); exit(1)
>> (althrough this will break mules
>> waiting on other lock.. there is also possibility that application use
>> multiple locks)
>>
>> sends kill
>>
>> Another (better) possible workaround is to use pthread_mutex and
>> pthread_mutexattr_setrobust with PTHREAD_MUTEX_ROBUST as uwsgi.lock.
>> (dont know about performance of rwlock vs pthread and portability )
>>
>> Hmm also there is SEM_UNDO. Dont know which is better pthread_mutex
>> of semaphore with SEM_UNDO. (Althrough it seems i can use semanchuk
>> sysv_ipc with SEM_UNDO right now on linux)
>>
>> ps.
>> Hmm according to Stevens mutexes is much faster that sem. need to
>> benchmark myself.
>>
>>
>
> I strongly hate sysv shared ipc :) Main reason is that each process can
> leave garbage in the system (garbage that sysadmins forget to clear 99% of
> the times). uwsgi.lock() (indpendently by the implementation) should work
> as expected as when a worker (a process) dies the lock is released.
>
> By the way enabling robust flag (in multithread modes) could be useful for
> third-party plugins wanting to spawn threads.
>
>
> --
> Roberto De Ioris
> http://unbit.it
> _______________________________________________
> uWSGI mailing list
> [email protected]
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
--
--------------------------------------------
Турнаев Евгений Викторович
+7 906 875 09 43
--------------------------------------------
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
