Hi All, I have a problem with uWSGI (2.0.12 on Ubuntu 14.04) - since some change in my python code (a few thousand lines of code were changed, its difficult understand now what caused the probelm) uwsgi does not reloads gracefully on SIGHUP as it were before - instead process detached only on 'harakiri' which is quite slow and all client connections are hang:
...gracefully killing workers... Gracefully killing worker 1 (pid: 2017)... Tue May 3 18:59:11 2016 - worker 1 (pid: 2017) is taking too much time to die...NO MERCY !!! worker 1 buried after 1 seconds binary reloading uWSGI... chdir() to /var/www/my_project/my_project closing all non-uwsgi socket fds > 2 (max_fd = 1024)... found fd 3 mapped to socket 0 (192.168.128.65:9991) running /var/www/my_project/my_project/env/bin/uwsgi [uWSGI] getting INI configuration from uwsgi-test.ini *** Starting uWSGI 2.0.12 (64bit) on [Tue May 3 18:59:12 2016] *** compiled with version: 4.8.4 on 03 May 2016 18:42:43 os: Linux-4.5.0-x86_64-linode65 #2 SMP Mon Mar 14 18:01:58 EDT 2016 nodename: lin-yp0-py2 machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 4 current working directory: /var/www/my_project/my_project detected binary path: /var/www/my_project/my_project/env/bin/uwsgi chdir() to /var/www/my_project/my_project/ your processes number limit is 15980 your memory page size is 4096 bytes *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) *** RRDtool library available at 0x1386500 *** uwsgi socket 0 inherited INET address 192.168.128.65:9991 fd 3 Python version: 2.7.6 (default, Jun 22 2015, 18:01:27) [GCC 4.8.2] Set PythonHome to /var/www/my_project/my_project/env *** Python threads support is disabled. You can enable it with --enable-threads *** Python main interpreter initialized at 0x139eaf0 your server socket listen backlog is limited to 300 connections your mercy for graceful operations on workers is 60 seconds mapped 145520 bytes (142 KB) for 1 cores *** Operational MODE: single process *** added /var/www/my_project/my_project/ to pythonpath. WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x139eaf0 pid: 1904 (default app) *** uWSGI is running in multiple interpreter mode *** gracefully (RE)spawned uWSGI master process (pid: 1904) spawned uWSGI worker 1 (pid: 2473, cores: 1) The problem happens in this line as far I see:
signature.asc
Description: PGP signature
Around the lines 12140 in function void gracefully_kill(int signum) {.
I did this to reproduce the problem: configured uwsgi to run 1 child
process and sent SIGHUP to 'master' process of UWSGI. On the child
process I ran gdb and here is backtrace:
(gdb) bt
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007ff4079876d8 in PyThread_acquire_lock () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2 0x00007ff407979f10 in ?? () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3 0x00007ff40799b0d4 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5 0x00007ff40799add8 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7 0x00007ff40799add8 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9 0x00007ff4079d17a5 in ?? () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007ff40793dd43 in PyObject_Call () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007ff4079973b1 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007ff4079d16d0 in ?? () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007ff40793dd43 in PyObject_Call () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007ff4079b6577 in PyEval_CallObjectWithKeywords () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007ff40791f8a7 in Py_Finalize () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x0000000000464481 in uwsgi_plugins_atexit () at core/uwsgi.c:1776
#18 0x00007ff40724c259 in __run_exit_handlers (status=status@entry=17,
listp=0x7ff4075ce6c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true)
at exit.c:82
#19 0x00007ff40724c2a5 in __GI_exit (status=status@entry=17) at exit.c:104
#20 0x000000000041ed9f in uwsgi_exit (status=status@entry=17) at
core/utils.c:4406
#21 0x0000000000465f0e in gracefully_kill (signum=<optimized out>) at
core/uwsgi.c:1240
#22 <signal handler called>
#23 0x00007ff40730ab13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#24 0x000000000045aa23 in event_queue_wait (eq=eq@entry=4, timeout=<optimized
out>, interesting_fd=interesting_fd@entry=0x7ffd1df47a1c) at core/event.c:720
#25 0x000000000041c5ae in wsgi_req_accept (queue=queue@entry=4,
wsgi_req=wsgi_req@entry=0x7ff408cf5078) at core/utils.c:1516
#26 0x00000000004634c6 in simple_loop_run (arg1=0x0) at core/loop.c:140
#27 0x00000000004675d4 in uwsgi_ignition () at core/uwsgi.c:3561
#28 0x000000000046bf5d in uwsgi_worker_run () at core/uwsgi.c:3489
#29 0x000000000046c54f in uwsgi_run () at core/uwsgi.c:3399
#30 0x000000000041b9fe in main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>) at core/uwsgi.c:2007
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85 in ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S
(gdb) bt
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007ff4079876d8 in PyThread_acquire_lock () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2 0x00007ff407979f10 in ?? () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3 0x00007ff40799b0d4 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5 0x00007ff40799add8 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7 0x00007ff40799add8 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9 0x00007ff4079d17a5 in ?? () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007ff40793dd43 in PyObject_Call () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007ff4079973b1 in PyEval_EvalFrameEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007ff40799c54d in PyEval_EvalCodeEx () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007ff4079d16d0 in ?? () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007ff40793dd43 in PyObject_Call () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007ff4079b6577 in PyEval_CallObjectWithKeywords () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007ff40791f8a7 in Py_Finalize () from
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x0000000000464481 in uwsgi_plugins_atexit () at core/uwsgi.c:1776
#18 0x00007ff40724c259 in __run_exit_handlers (status=status@entry=17,
listp=0x7ff4075ce6c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true)
at exit.c:82
#19 0x00007ff40724c2a5 in __GI_exit (status=status@entry=17) at exit.c:104
#20 0x000000000041ed9f in uwsgi_exit (status=status@entry=17) at
core/utils.c:4406
#21 0x0000000000465f0e in gracefully_kill (signum=<optimized out>) at
core/uwsgi.c:1240
#22 <signal handler called>
#23 0x00007ff40730ab13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#24 0x000000000045aa23 in event_queue_wait (eq=eq@entry=4, timeout=<optimized
out>, interesting_fd=interesting_fd@entry=0x7ffd1df47a1c) at core/event.c:720
#25 0x000000000041c5ae in wsgi_req_accept (queue=queue@entry=4,
wsgi_req=wsgi_req@entry=0x7ff408cf5078) at core/utils.c:1516
#26 0x00000000004634c6 in simple_loop_run (arg1=0x0) at core/loop.c:140
#27 0x00000000004675d4 in uwsgi_ignition () at core/uwsgi.c:3561
#28 0x000000000046bf5d in uwsgi_worker_run () at core/uwsgi.c:3489
#29 0x000000000046c54f in uwsgi_run () at core/uwsgi.c:3399
#30 0x000000000041b9fe in main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>) at core/uwsgi.c:2007
(gdb)
I tried those:
no-threads-wait=true
enable-threads=false
but it does not help.
What else should I do to find out what causes the process to hang on
exit() ?
Best Regards,
Alex
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
