Hi there uWSGI fans,
We have an emperor setup, with multiple vassals that run in chroots. Each
vassal has a master and one or many workers. They listen on sockets which,
thanks to a little symlink-based magic, are available from inside the chroot
and outside.
We think we’ve got a race condition during vassal reloads, and were wondering
if anyone could help confirm or deny our theory?
When we do a reload, we see output that looks a little like this:
Wed Aug 21 16:55:19 2013 - received message 1 from emperor
...gracefully killing workers...
Gracefully killing worker 1 (pid: 1881)...
worker 1 buried after 1 seconds
binary reloading uWSGI...
chdir() to /etc/uwsgi/vassals
chdir(): No such file or directory [core/master_utils.c line 348]
closing all non-uwsgi socket fds > 2 (max_fd = 123456)...
found fd 3 mapped to socket 0 (/var/sockets/admin.pythonanywhere.com/socket)
running /usr/bin/uwsgi
execvp(): No such file or directory [core/master_utils.c line 449]
chdir(): No such file or directory [core/uwsgi.c line 1326]
VACUUM: unix socket /var/sockets/admin.pythonanywhere.com/socket removed.
limiting number of processes to 32...
your processes number limit is 32
your memory page size is 4096 bytes
detected max file descriptor number: 123456
building mime-types dictionary from file /etc/mime.types...532 entry found
lock engine: pthread robust mutexes
uwsgi socket 0 bound to UNIX address
/var/sockets/admin.pythonanywhere.com/socket fd 3
Python version: 2.7.4 (default, Apr 19 2013, 18:30:41) [GCC 4.7.3]
*** Python threads support is disabled. You can enable it with
--enable-threads ***
Python main interpreter initialized at 0xd5f180
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
setting request body buffering size to 65536 bytes
mapped 333952 bytes (326 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xd5f180 pid:
1896 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1896)
spawned uWSGI worker 1 (pid: 1901, cores: 1)
announcing my loyalty to the Emperor...
I’ve separated it out into 3 sections, according to what we think is going on.
In the first part is a message about uWSGI killing its worker(s)
The second part seems to be a *master* vassal attempting to respawn, but it
fails to do so because it’s already in the chroot, where paths like
/etc/uwsgi/vassals, and /usr/bin/uwsgi are not available. It gives up, and in
the process, deletes the socket.
The third part is the normal output we see when we start up a vassal for the
first time, so we think it’s a normal vassal re-start, and we see the master
pid and worker pid coming up tidily.
THE PROBLEM: is that we think there’s a race between the spurious respawn in
part 2 which deletes the socket, and the desired restart in part 3, and we
think that sometimes the guy in part 2 manages to delete the socket after part
3 has started, which leaves you with a broken vassal.
QUESTION: is this even possible? Are we right about what’s going on in part
2? What should we do about it?
Thanks in advance for any suggestions!
Harry + the PythonAnywhere gang.
--
Harry Percival
Developer
[email protected]
PythonAnywhere - a fully browser-based Python development and hosting
environment
<http://www.pythonanywhere.com/>
PythonAnywhere LLP
17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79
Registered in England and Wales as company number OC378414.
Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
