Hello, today we´ve upgraded one of our loadbalancers to linux kernel 3.7.1 and haproxy 1.5 dev 17 - coming from kernel 3.0.1 and haproxy 1.5 dev 7.
After the upgrade, the system is delivering traffic as usual and we don´t see any traffic issues. But now there are constantly 4 CPUs 100% busy with about 30 % user and 70 % system load. We haven´t seen this behaviour in the past. Additionally we get the following messages every couple of minutes: Jan 8 18:30:59 srv11 kernel: [ 3878.272003] ------------[ cut here ]------------ Jan 8 18:30:59 srv11 kernel: [ 3878.295572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 8 18:30:59 srv11 kernel: [ 3878.319107] Hardware name: System x3690 X5 -[7148Z68]- Jan 8 18:30:59 srv11 kernel: [ 3878.340686] cleanup rbuf bug: copied 7B02E4F6 seq 7B01F558 rcvnxt 7B02E4F6 Jan 8 18:30:59 srv11 kernel: [ 3878.363160] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat acpi_cpufreq snd_pcm cdc_ether usbnet mii coretemp kvm_intel kvm snd_timer snd crc32c_intel evdev joydev hid_generic soundcore microcode snd_page_alloc serio_raw pcspkr mperf tpm_tis processor ioatdma lpc_ich i2c_i801 tpm shpchp mfd_core tpm_bios pci_hotplug i2c_core dca thermal_sys button ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif ata_generic usbhid hid uhci_hcd ata_piix libata megaraid_sas ehci_hcd bnx2 usbcore scsi_mod usb_common be2net Jan 8 18:30:59 srv11 kernel: [ 3878.513462] Pid: 30307, comm: haproxy Tainted: G W 3.7.1 #1 Jan 8 18:30:59 srv11 kernel: [ 3878.540064] Call Trace: Jan 8 18:30:59 srv11 kernel: [ 3878.564947] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c Jan 8 18:30:59 srv11 kernel: [ 3878.591417] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a Jan 8 18:30:59 srv11 kernel: [ 3878.617965] [<ffffffff812d3e02>] ? tcp_cleanup_rbuf+0x4d/0xfc Jan 8 18:30:59 srv11 kernel: [ 3878.645352] [<ffffffff812d4034>] ? tcp_read_sock+0x183/0x194 Jan 8 18:30:59 srv11 kernel: [ 3878.670621] [<ffffffff812d487d>] ? tcp_sendpage+0x45b/0x45b Jan 8 18:30:59 srv11 kernel: [ 3878.696935] [<ffffffff812d4118>] ? tcp_splice_read+0xd3/0x223 Jan 8 18:30:59 srv11 kernel: [ 3878.721845] [<ffffffff8112d9ae>] ? sys_splice+0x345/0x3bf Jan 8 18:30:59 srv11 kernel: [ 3878.746239] [<ffffffff813651a9>] ? system_call_fastpath+0x16/0x1b Jan 8 18:30:59 srv11 kernel: [ 3878.770749] ---[ end trace 91a60bafa2f9d85e ]--- This is our global configuration and one of the most busy threads (about 1k requests/s): global daemon maxconn 131072 spread-checks 2 stats socket /var/run/haproxy.sock nbproc 34 defaults mode http option splice-response option splice-request timeout connect 5000ms timeout client 30000ms timeout server 300000ms timeout http-request 20000ms # option forceclose frontend marketing-in bind <ip>:80 default_backend marketing maxconn 32768 option http-server-close option forwardfor reqidel ^X-Forwarded-For:.* bind-process 17 backend marketing stats enable stats uri <uri> stats auth <user> option httpchk GET /server_up.php http-check expect rstring ^OK$ balance roundrobin server web1 <ip1>:80 maxconn 4096 check port 80 inter 10000 fastinter 2000 server web2 <ip2>:80 maxconn 4096 check port 80 inter 10000 fastinter 2000 server web3 <ip3>:80 maxconn 4096 check port 80 inter 10000 fastinter 2000 server web4 <ip4>:80 maxconn 4096 check port 80 inter 10000 fastinter 2000 server sorry <sorry>:80 check backup Additionally this are the build options before and now: HA-Proxy version 1.5-dev7 2011/09/10 Copyright 2000-2011 Willy Tarreau <[email protected]> Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -march=core2 -m64 OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. Note: This is not dev17 anymore - this is todays snapshot HA-Proxy version 1.5-dev17 2012/12/28 Copyright 2000-2012 Willy Tarreau <[email protected]> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -march=core2 -m64 OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3.4 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 0.9.8o 01 Jun 2010 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Do you have any idea what´s causing this issues? Thank you very much in advance! Regards, Christian ____________________________ Christian Becker Systemadministration Travian Games GmbH Wilhelm-Wagenfeld-Str. 22 80807 München Germany Tel.: +49 / (0)89 / 324 915 – 0 Fax: +49 / (0)89 / 324 915 – 970 [email protected] www.traviangames.de Sitz der Gesellschaft München AG München HRB: 173511 Geschäftsführer: Siegfried Müller USt-IdNr.: DE246258085 Diese Email einschließlich ihrer Anlagen ist vertraulich und nur für den Adressaten bestimmt. Wenn Sie nicht der vorgesehene Empfaenger sind, bitten wir Sie, diese Email mit Anlagen unverzueglich und vollstaendig zu loeschen und uns umgehend zu benachrichtigen. This email and its attachments are strictly confidential and are intended solely for the attention of the person to whom it is addressed. If you are not the intended recipient of this email, please delete it including its attachments immediately and inform us accordingly.
