> Jan 8 18:30:59 srv11 kernel: [ 3878.272003] ------------[ cut here > ]------------ > Jan 8 18:30:59 srv11 kernel: [ 3878.295572] WARNING: at net/ipv4/tcp.c:1330 > tcp_cleanup_rbuf+0x4d/0xfc() > Jan 8 18:30:59 srv11 kernel: [ 3878.319107] Hardware name: System x3690 X5 > -[7148Z68]- > Jan 8 18:30:59 srv11 kernel: [ 3878.340686] cleanup rbuf bug: copied > 7B02E4F6 seq 7B01F558 rcvnxt 7B02E4F6 > Jan 8 18:30:59 srv11 kernel: [ 3878.363160] Modules linked in: 8021q garp > stp llc nls_utf8 nls_cp437 vfat fat acpi_cpufreq snd_pcm cdc_ether usbnet mii > coretemp kvm_intel kvm snd_timer snd crc32c_intel evdev joydev hid_generic > soundcore microcode snd_page_alloc serio_raw pcspkr mperf tpm_tis processor > ioatdma lpc_ich i2c_i801 tpm shpchp mfd_core tpm_bios pci_hotplug i2c_core > dca thermal_sys button ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod > crc_t10dif ata_generic usbhid hid uhci_hcd ata_piix libata megaraid_sas > ehci_hcd bnx2 usbcore scsi_mod usb_common be2net > Jan 8 18:30:59 srv11 kernel: [ 3878.513462] Pid: 30307, comm: haproxy > Tainted: G W 3.7.1 #1 > Jan 8 18:30:59 srv11 kernel: [ 3878.540064] Call Trace: > Jan 8 18:30:59 srv11 kernel: [ 3878.564947] [<ffffffff8103ef70>] ? > warn_slowpath_common+0x78/0x8c > Jan 8 18:30:59 srv11 kernel: [ 3878.591417] [<ffffffff8103f023>] ? > warn_slowpath_fmt+0x45/0x4a > Jan 8 18:30:59 srv11 kernel: [ 3878.617965] [<ffffffff812d3e02>] ? > tcp_cleanup_rbuf+0x4d/0xfc > Jan 8 18:30:59 srv11 kernel: [ 3878.645352] [<ffffffff812d4034>] ? > tcp_read_sock+0x183/0x194 > Jan 8 18:30:59 srv11 kernel: [ 3878.670621] [<ffffffff812d487d>] ? > tcp_sendpage+0x45b/0x45b > Jan 8 18:30:59 srv11 kernel: [ 3878.696935] [<ffffffff812d4118>] ? > tcp_splice_read+0xd3/0x223 > Jan 8 18:30:59 srv11 kernel: [ 3878.721845] [<ffffffff8112d9ae>] ? > sys_splice+0x345/0x3bf > Jan 8 18:30:59 srv11 kernel: [ 3878.746239] [<ffffffff813651a9>] ? > system_call_fastpath+0x16/0x1b > Jan 8 18:30:59 srv11 kernel: [ 3878.770749] ---[ end trace 91a60bafa2f9d85e > ]---
I recall we had a similar report on the ML back in September with a 3.5.0 kernel (thread @[1]). Like the old report your one shows a tainted kernel as well. However both reports have the G and the W flag, that probably means the kernel is only "Tainted" because its not the first kernel warning since boot (first warning sets WARN_ON - the W flag; G means everything in your kernel is GPL or GPL compliant). Can you confirm you are running a vanilla kernel from kernel.org? If my suspicion is correct, you should find the same kernel warning in the beginning of your dmesg, but without the "Tainted" sign. I suggest you post that kernel warning on the linux netdev mailing list, as Willy suggested in the older thread. > After removing the option splice-response and option splice-request the cpu > load instantly dropped to a almost normal level. > So i guess there are either in haproxy or in the kernel some splice issues. In fact, this seems to be a kernel issue with TCP splicing. > i´ll try to downgrade the kernel tomorrow to 3.7 or maybe lower. I believe the regression is older than that. If you have the possibility, you could give 3.4.24 and 3.5.7 a try. If 3.4.24 works and 3.5.7 doesn't, then the regression has been introduced with the 3.5 kernel. [1] http://permalink.gmane.org/gmane.comp.web.haproxy/9560 Regards, Lukas
