Re: [uWSGI] gevent worker graceful reload during rush hour?

Mon, 25 Dec 2017 01:26:42 -0800 

Hi,

You seem to be using the gevent loop handler.

What value are you using to produce this problem?

How many CPU cores do you have?

Regards,

Etienne


Le 2017-12-24 à 20:54, est a écrit :

Hi,
uWSGi was running great for several months until last night during Xmas event 


...The work of process 6442 is done. Seeya!
Mon Dec 25 00:01:06 2017 - Fire in the hole !!! (60 seconds to detonation)
Gracefully killing worker 3 (pid: 6442)...
Mon Dec 25 00:01:06 2017 - stopping gevent signals watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - stopping gevent sockets watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - main gevent watchers stopped for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 118 is managing "POST " for Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 119 is managing "POST " for Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 123 is managing "POST " for Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 127 is managing "POST " for Mon Dec 25 00:01:06 2017 - waiting for 4 running requests on worker 3 (pid: 6442)... 
...The work of process 6442 is done. Seeya!
Mon Dec 25 00:01:06 2017 - Fire in the hole !!! (60 seconds to detonation)
Gracefully killing worker 3 (pid: 6442)...
Mon Dec 25 00:01:06 2017 - stopping gevent signals watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - stopping gevent sockets watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - main gevent watchers stopped for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 118 is managing "POST " for Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 123 is managing "POST " for Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 127 is managing "POST " for Mon Dec 25 00:01:06 2017 - waiting for 3 running requests on worker 3 (pid: 6442)... 
...The work of process 6442 is done. Seeya!
Mon Dec 25 00:01:06 2017 - Fire in the hole !!! (60 seconds to detonation)
Gracefully killing worker 3 (pid: 6442)...
Mon Dec 25 00:01:06 2017 - stopping gevent signals watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - stopping gevent sockets watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - main gevent watchers stopped for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 118 is managing "POST " for Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 127 is managing "POST " for Mon Dec 25 00:01:06 2017 - waiting for 2 running requests on worker 3 (pid: 6442)... 
...The work of process 6442 is done. Seeya!
Mon Dec 25 00:01:06 2017 - Fire in the hole !!! (60 seconds to detonation)
Gracefully killing worker 3 (pid: 6442)...
Mon Dec 25 00:01:06 2017 - stopping gevent signals watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - stopping gevent sockets watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - main gevent watchers stopped for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - worker 3 (pid: 6442) core 127 is managing "POST " for Mon Dec 25 00:01:06 2017 - waiting for 1 running requests on worker 3 (pid: 6442)... 
...The work of process 6442 is done. Seeya!
Mon Dec 25 00:01:06 2017 - Fire in the hole !!! (60 seconds to detonation)
Gracefully killing worker 3 (pid: 6442)...
Mon Dec 25 00:01:06 2017 - stopping gevent signals watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - stopping gevent sockets watchers for worker 3 (pid: 6442)... Mon Dec 25 00:01:06 2017 - main gevent watchers stopped for worker 3 (pid: 6442)... 
goodbye to the gevent Hub on worker 3 (pid: 6442)
worker 3 killed successfully (pid: 6442)
Respawned uWSGI worker 3 (new pid: 7656)
mapping worker 3 to CPUs: 2
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x94dbb0 pid: 7656 (default app) Mon Dec 25 00:01:10 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (101/100) *** Mon Dec 25 00:01:11 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (101/100) *** Mon Dec 25 00:01:13 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (101/100) *** Mon Dec 25 00:01:16 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (100/100) *** Mon Dec 25 00:01:22 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (100/100) *** Mon Dec 25 00:01:26 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (101/100) *** Mon Dec 25 00:01:27 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (100/100) *** Mon Dec 25 00:01:28 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (100/100) *** Mon Dec 25 00:01:30 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (101/100) *** Mon Dec 25 00:01:33 2017 - *** uWSGI listen queue of socket ":9000" (fd: 6) full !!! (100/100) *** 

What happened to this?

Is my listening backlog too low, or the worker is not quite ready?
Can we spawn a new worker and get ready first, then kill the old worker next? 



_______________________________________________
uWSGI mailing list
uWSGI@lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi


--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/


_______________________________________________
uWSGI mailing list
uWSGI@lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Reply via email to