Hi, I've got a pretty weird situation here and I can't figure out what's going on.
My application is a simple WSGI app which uses Kazoo (https://kazoo.readthedocs.org/en/latest/) to stay connected to a Zookeeper server and dynamically update the app's configuration when the data changes. The problem I'm facing is the following: if I run the application from the command line (as root), uWSGI and the app start and behave properly, but whenever I start the application using an Upstart script (I'm on Ubuntu 12.04) it hangs at some point after starting up and it's not answering to requests. >From what I know, Kazoo uses a background thread to talk to the zookeeper server. This one produces periodic debug messages "sent ping", "received ping" in my kazoo version so I know that when I see these messages, the application is working. When it's hang though, the log *looks* like this thread has not even started (that's not true, see below). So, what's the difference between running the application under upstart and from the command line? Why is it working in one case and not in the other? To me it looks like uWSGI deadlocks for a reason unknown to me. Technical details ========== * Versions - Ubuntu: 12.04 - Upstart: 1.5-0ubuntu7.2 - uwsgi: 1.0.3+dfsg-1ubuntu0.1 (ubuntu package) , 1.9.21 (in-house built .deb) or 2.0.5.1 (pip install uwsgi) - problem persists with all of them - python: 2.7.3-0ubuntu2.2 - python-kazoo: 1.3.1-1 (in-house built .deb) or 2.0 (pip install kazoo) - python-bottle: 0.12.0 (in-house built .deb) * Platform: EC2 Ubuntu 12.04 instance * Application (needs kazoo and bottle): test.py, save as /home/ubuntu/uwsgi/test.py: http://pastebin.com/w0B6WMat wsgi.xml, save as /home/ubuntu/uwsgi/wsgi.xml: http://pastebin.com/fZmiUZCY * How to test: $ curl http://localhost:9090/ping , if it hangs, it's broken. * uWSGI configuration: I'm running a setup with one master process and a single worker, threads enabled (see wsgi.xml above). Running from the command line: /usr/bin/uwsgi_python --xmlconfig /home/ubuntu/uwsgi/wsgi.xml --http-socket 0.0.0.0:9090 Running from upstart (config file): http://pastebin.com/9HrAaZxP * Debugging If i remove the <lazy /> option from the config file or if I replace it with <lazy-app /> the application only works when started from the command line. If started from Upstart, it deadlocks and doesn't even create the worker process, nor does it change the process title as specified in the config file. So removing <lazy> makes things worse. ----[ application hang (running in Upstart) 1) attached gdb to the worker process (gdb) info thread Id Target Id Frame 4 Thread 0x7f87c1ae0700 (LWP 4990) "uwsgi_python" 0x00007f87c8db6fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 3 Thread 0x7f87c12df700 (LWP 4991) "uwsgi_python" 0x00007f87c8db6fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x7f87c0ade700 (LWP 4992) "uwsgi_python" 0x00007f87c8db6fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x7f87c91d0780 (LWP 4985) "uwsgi_python" 0x00007f87c8db6fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 2) gdb 'thread apply all where' : http://pastebin.com/z5USTQGW 3) log file: http://pastebin.com/eyRujTBh ----[ application working (running from commandline) 1) attached gdb to the worker process (gdb) info thread Id Target Id Frame 4 Thread 0x7f94fc391700 (LWP 5630) "uwsgi_python" 0x00007f950362afd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 3 Thread 0x7f94fbb90700 (LWP 5631) "uwsgi_python" 0x00007f950362afd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x7f94fb38f700 (LWP 5632) "uwsgi_python" 0x00007f9501f95763 in select () from /lib/x86_64-linux-gnu/libc.so.6 * 1 Thread 0x7f9503a44780 (LWP 5625) "uwsgi_python" 0x00007f9501f9ca93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 2) gdb 'thread apply all where': http://pastebin.com/r4iMB9Za 3) log file: http://pastebin.com/ybrMjVU6 GDB backtraces are against uwsgi 1.0.3 , log files are from 1.9.21. However, there seems to be no difference between the versions with respect to this issue. Any ideas? Regards, Cosmin _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
