2 apps are using uwsgi 2.0.11.1 and one 2.0.9. two of them (different two)
use uwsgi http workers and one uses buffered uwsgi
configurations attached beneath.
Suddenly, since a few days ago, we get a strange (and bad) condition in our
AWS app server: All uwsgi process cpu usage sky rockets to 100%. and stuck
there, not processing requests and fail to stop unless I kill them.
I read about the thundering-herd condition in the excellent article in
uwsgi site but I don't think this applies to our condition:
1. All three apps cpu skyrockets together.
2. If my understanding is correct then thundering-herd cpu usage would
eventually go down, that doesn't happen in my case
When I look with strace on a process for example I get lots of this:
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
futex(0x703a68, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
futex(0x703a68, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
epoll_wait(4, {{EPOLLIN|EPOLLHUP, {u32=0, u64=0}}}, 1, -1) = 1
accept4(0, 0x7ffb69ed60a2, [110], SOCK_NONBLOCK) = -1 EINVAL (Invalid
argument)
futex(0x703a68, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x703a68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
I would be glad for any help or clues about that!
Thanks
Example configuration for different apps
[uwsgi]
socket = /tmp/rest.sock
chmod-socket = 664
uid = www-data
gid = www-data
master = true
processes = 4
threads = 25
vaccum = true
max-requests = 15000
harakiri = 30
die-on-term = true
Second app:
http = 0.0.0.0:9006
master = true
processes = 2
threads = 10
vaccum = true
max-requests = 5000
harakiri = 20
die-on-term = true
Third app very similar to the second
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
