Hi,
I'm attempting to gracefully deploy code update on Django app by
forking vassal's master process.
We have long-running http sessions, so "r" reload was not graceful.
Due to memory consumption, we avoid lazy-apps. So, we use pre-forking
mode, which rules out pure worker reload ("w" and "c"). What I'm
trying next is fork reload "f" without Zerg. Fortunately, we don't
have any issues with conflicting pid/log files. Thou, I'm hitting
something else..
Here is my Vassal config (inspired by "The Art of Graceful Reloading"):
[uwsgi]
master-fifo = /home/foo/var/uwsgi/%n.new.fifo
master-fifo = /home/foo/var/uwsgi/%n.master.fifo
master = true
lazy-apps = false
enable-threads = true
thunder-lock = true
processes = 4
socket = /home/foo/var/uwsgi/%n.sock
chmod-socket = 666
vacuum = true
worker-reload-mercy = 600
if-exists = /home/foo/var/uwsgi/%n.master.fifo
hook-accepting1-once = writefifo:/home/foo/var/uwsgi/%n.master.fifo q
endif =
hook-accepting1-once = writefifo:/home/foo/var/uwsgi/%n.new.fifo 1
Emperor (running from systemd):
uwsgi --emperor %h/.config/uwsgi/vassal
After triggering master fork..
echo f > /home/foo/var/uwsgi/helloworld.master.fifo
..this is what happens:
* new master is forked and app loaded
* new workers are forked
* old master is triggered to shutdown
* then this appears in log - PROBLEM
uwsgi_master_manage_emperor()/read(): Resource temporarily unavailable
[core/emperor.c line 2424]
lost connection with my emperor !!!
SIGINT/SIGQUIT received...killing workers...
* new master will kill its workers and shutdown itself
* emperor forks another master (3rd one)
If in the hook, I "pause" the old master instead of shutdown "q", it
works. It appears as if new master won't survive the death of his
parent. I have also tested it with three states new/running/sleeping.
New master will survive first iteration (parent is sleeping), but not
the second (parent is shut down).
I did test it also with "vacuum = false" with no noticeable effect on the issue.
Q: is this scenario supported ? Forking master and shutting down his
parent.. In Emperor setup.
Thanks,
BranoZ
Full log from forking reload:
Thu Jan 28 18:35:00 2016 - new master copy spawned with pid 5311
fork()'ing uWSGI...
chdir() to /home/foo/.config/uwsgi/vassal
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (/home/foo/var/uwsgi/helloworld.sock)
running /home/foo/venv/uwsgi/bin/uwsgi
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from helloworld.ini
*** Starting uWSGI 2.0.12 (64bit) on [Thu Jan 28 18:35:00 2016] ***
compiled with version: 4.9.2 on 28 January 2016 15:37:16
os: Linux-3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
nodename: debian.example.com
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/foo/.config/uwsgi/vassal
detected binary path: /home/foo/venv/uwsgi/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/foo/helloworld
your processes number limit is 3934
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 inherited UNIX address /home/foo/var/uwsgi/helloworld.sock fd 3
Python version: 2.7.9 (default, Mar 1 2015, 13:01:26) [GCC 4.9.2]
Set PythonHome to /home/foo/venv/helloworld
Python main interpreter initialized at 0x1a4ffc0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 600 seconds
mapped 363840 bytes (355 KB) for 4 cores
*** Operational MODE: preforking ***
Loading helloworld/settings.py
Importing helloapp/__init__.py
pid 5311: Slowly starting app.. 1/10
pid 5311: Slowly starting app.. 2/10
pid 5311: Slowly starting app.. 3/10
pid 5311: Slowly starting app.. 4/10
pid 5311: Slowly starting app.. 5/10
pid 5311: Slowly starting app.. 6/10
pid 5311: Slowly starting app.. 7/10
pid 5311: Slowly starting app.. 8/10
pid 5311: Slowly starting app.. 9/10
pid 5311: Slowly starting app.. 10/10
Importing helloapp/apps.py
Importing helloapp/models.py
Running ready() on helloapp
WSGI app 0 (mountpoint='') ready in 11 seconds on interpreter
0x1a4ffc0 pid: 5311 (default app)
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 5311)
spawned uWSGI worker 1 (pid: 5453, cores: 1)
spawned uWSGI worker 2 (pid: 5454, cores: 1)
spawned uWSGI worker 3 (pid: 5455, cores: 1)
spawned uWSGI worker 4 (pid: 5456, cores: 1)
Thu Jan 28 18:35:11 2016 - [emperor] vassal helloworld.ini is ready to
accept requests
running "writefifo:/home/foo/var/uwsgi/helloworld.master.fifo q"
(accepting1-once)...
Thu Jan 28 18:35:11 2016 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 5088)...
Gracefully killing worker 2 (pid: 5089)...
Gracefully killing worker 3 (pid: 5090)...
Gracefully killing worker 4 (pid: 5091)...
running "writefifo:/home/foo/var/uwsgi/helloworld.new.fifo 1"
(accepting1-once)...
Thu Jan 28 18:35:11 2016 - active master fifo is now
/home/foo/var/uwsgi/helloworld.master.fifo
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
goodbye to uWSGI.
VACUUM: unix socket /home/foo/var/uwsgi/helloworld.sock removed.
Thu Jan 28 18:35:14 2016 - [emperor] removed uwsgi instance helloworld.ini
uwsgi_master_manage_emperor()/read(): Resource temporarily unavailable
[core/emperor.c line 2424]
lost connection with my emperor !!!
SIGINT/SIGQUIT received...killing workers...
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from helloworld.ini
*** Starting uWSGI 2.0.12 (64bit) on [Thu Jan 28 18:35:14 2016] ***
compiled with version: 4.9.2 on 28 January 2016 15:37:16
os: Linux-3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
nodename: debian.example.com
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/foo/.config/uwsgi/vassal
detected binary path: /home/foo/venv/uwsgi/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/foo/helloworld
your processes number limit is 3934
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to UNIX address /home/foo/var/uwsgi/helloworld.sock fd 3
Python version: 2.7.9 (default, Mar 1 2015, 13:01:26) [GCC 4.9.2]
Set PythonHome to /home/foo/venv/helloworld
Python main interpreter initialized at 0x1d16ff0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 600 seconds
mapped 363840 bytes (355 KB) for 4 cores
*** Operational MODE: preforking ***
Loading helloworld/settings.py
Importing helloapp/__init__.py
pid 5504: Slowly starting app.. 1/10
pid 5504: Slowly starting app.. 2/10
VACUUM WARNING: unix socket /home/foo/var/uwsgi/helloworld.sock
changed inode. Skip removal
pid 5504: Slowly starting app.. 3/10
pid 5504: Slowly starting app.. 4/10
pid 5504: Slowly starting app.. 5/10
pid 5504: Slowly starting app.. 6/10
pid 5504: Slowly starting app.. 7/10
pid 5504: Slowly starting app.. 8/10
pid 5504: Slowly starting app.. 9/10
pid 5504: Slowly starting app.. 10/10
Importing helloapp/apps.py
Importing helloapp/models.py
Running ready() on helloapp
WSGI app 0 (mountpoint='') ready in 10 seconds on interpreter
0x1d16ff0 pid: 5504 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 5504)
Thu Jan 28 18:35:24 2016 - [emperor] vassal helloworld.ini has been spawned
spawned uWSGI worker 1 (pid: 5627, cores: 1)
spawned uWSGI worker 2 (pid: 5628, cores: 1)
spawned uWSGI worker 3 (pid: 5629, cores: 1)
spawned uWSGI worker 4 (pid: 5630, cores: 1)
Thu Jan 28 18:35:24 2016 - [emperor] vassal helloworld.ini is ready to
accept requests
running "writefifo:/home/foo/var/uwsgi/helloworld.master.fifo q"
(accepting1-once)...
open("/home/foo/var/uwsgi/helloworld.master.fifo"): No such device or
address [core/hooks.c line 134]
running "writefifo:/home/foo/var/uwsgi/helloworld.new.fifo 1"
(accepting1-once)...
Thu Jan 28 18:35:24 2016 - active master fifo is now
/home/foo/var/uwsgi/helloworld.master.fifo
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
