Hi Willy,
I've been upgraded our server to Ubuntu 14.04 LTS:
root@WD-G0-SRP1:~# uname -a
Linux WD-G0-SRP1 3.13.0-70-generic #113-Ubuntu SMP Mon Nov 16 18:34:13 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
And this problem appeared again. HAProxy -vv:
HA-Proxy version 1.6.2 2015/11/03
Copyright 2000-2015 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3.4
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
I caught both strace log and show sess all output this time:
http://baiy.cn/tmp/log-1124.rar
Thanks :-)
--
Best Regards
BaiYang
baiy...@gmail.com
http://baiy.cn
**** < END OF EMAIL > ****
From: Willy Tarreau
Date: 2015-11-22 01:55
To: baiyang
CC: Lukas Tribus; haproxy
Subject: Re: Re: CPU 100% when waiting for the client timeout
Hi Baiyang,
On Sun, Nov 22, 2015 at 01:22:42AM +0800, baiyang wrote:
> Hi Willy,
>
> It reproduced again:
Ah bad :-/
(...)
> I've captured the strace call log successfully this time. You are right,
> there are full of epoll_wait and gettimeofday calls.
Excellent, thanks for doing this.
> Here is the broken request:
> Nov 22 01:06:32 WD-G0-SRP1 haproxy[1259]: 2.145.41.3:56014
> [22/Nov/2015:00:50:01.683] https-in~ g0n2/n2 1263/0/0/3/991286 200 84517 - -
> CD-- 8/8/1/1/0 0/0 "GET /lib/ext/ext-all.js HTTP/1.1"
>
> I have been uploaded the total 40 seconds strace log and full haproxy.log to:
> http://baiy.cn/tmp/log-1122.rar
OK I just downloaded it, thank you. That's interesting, it's not a
closed fd that is reported, it's an old timer that is not processed
(for a reason I have no idea about) or a task which remains active
but not processed, which causes epoll_wait's timeout to remain zero,
hence the busy loop. This clearly is an haproxy bug.
Did you by chance have the opportunity to retrieve a "show sess all"
on the CLI ? I can easily understand that you didn't think about it
facing the situation.
At least that gives us a few insights about what to look for :-/
Thanks!
willy
