The following reply was made to PR kernel/6486; it has been noted by GNATS.
From: Claudio Jeker <[email protected]> To: Anton Maksimenkov <[email protected]> Cc: [email protected] Subject: Re: kernel/6486: MCLGETI breaks re(4) Date: Wed, 13 Oct 2010 09:58:49 +0200 On Wed, Oct 13, 2010 at 11:44:01AM +0600, Anton Maksimenkov wrote: > >Number: 6486 > >Category: kernel > >Synopsis: MCLGETI breaks re(4) > >Confidential: yes > >Severity: serious > >Priority: medium > >Responsible: bugs > >State: open > >Quarter: > >Keywords: > >Date-Required: > >Class: sw-bug > >Submitter-Id: unknown > >Arrival-Date: Wed Oct 13 05:50:01 GMT 2010 > >Closed-Date: > >Last-Modified: > >Originator: > >Release: > >Organization: > >Environment: > System : OpenBSD 4.8 > Details : OpenBSD 4.8-current (GENERIC) #3: Thu Oct 7 19:25:57 > YEKST 2010 > > [email protected]:/usr/src/sys/arch/i386/compile/GENERIC > > Architecture: OpenBSD.i386 > Machine : i386 > >Description: > When system reaches thousand connections, machine becomes unresponsive. > Though it's possible to break into ddb (stack traces differ from time > to time). > > Original report: http://marc.info/?l=openbsd-misc&m=128426958003630&w=2 > > >How-To-Repeat: > Run any network stress program that about a thousand connections. > > >Fix: > Use pre-MCLGETI version of the driver. > I ran tcpbench -n 1000 on my box with re(4) and it worked without any hangs. Did you read my mail about increasing kern.maxclusters? http://marc.info/?l=openbsd-misc&m=128435828008471&w=2 >From your original report: ddb> show all pools Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle ... mcl2k 2048 56587 2451736 54237 1175 0 1175 1175 4 3072 0 mbpl 256 2581219 0 2577872 211 0 211 211 1 384 1 ... You ended up with 2451736 failed mcluster allocations. This is the reason why your network is stuck. Most probably re(4) was unable to allocate clusters and now the RX ring is empty and stuck, in the worst case you hit an interrupt storm. >From my test: netstat -m 20/128034/614400 mbuf 2048 byte clusters in use (current/peak/max) As you can see during my tcpbench test I peaked at 128034 active clusters which is way more then the 6144 setup by default. Oh and just for the kicks: Memory resource pool statistics Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle mbpl 256 68895154 10 52 9645 0 9645 9645 1 38400 9641 mcl2k 2048 7987962 11 20 64017 0 64017 64017 4 307200 64006 As you can see my i386 had mbuf and mcluster failures because I run the kernel out of kvm (this would not have happend if I increased kern.maxclusters a bit more carefully). Anyway, you need to properly tune your system to handle 1000 and more TCP connections. -- :wq Claudio OpenBSD 4.8-current (GENERIC.MP) #0: Thu Sep 30 18:10:06 CEST 2010 [email protected]:/usr/src/sys/arch/i386/compile/GENERIC.MP cpu0: AMD Phenom(tm) II X6 1055T Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 2.81 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT real mem = 3486650368 (3325MB) avail mem = 3419586560 (3261MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 05/04/10, BIOS32 rev. 0 @ 0xf0010, SMBIOS rev. 2.6 @ 0x9f400 (70 entries) bios0: vendor American Megatrends Inc. version "1456" date 05/04/2010 bios0: ASUSTeK Computer INC. M4A89GTD-PRO/USB3 acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S3 S4 S5 acpi0: tables DSDT FACP APIC MCFG OEMB SRAT HPET SSDT acpi0: wakeup devices PCE2(S4) PCE3(S4) PCE4(S4) PCE5(S4) PCE6(S4) PCE7(S4) PCE9(S4) PCEA(S4) PCEB(S4) PCEC(S4) SBAZ(S4) P0PC(S4) UHC1(S4) UHC2(S4) USB3(S4) UHC4(S4) USB5(S4) UHC6(S4) UHC7(S4) PE20(S4) PE21(S4) PE22(S4) PE23(S4) PS2K(S4) PS2M(S4) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 200MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: AMD Phenom(tm) II X6 1055T Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 2.81 GHz cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT cpu2 at mainbus0: apid 2 (application processor) cpu2: AMD Phenom(tm) II X6 1055T Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 2.81 GHz cpu2: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT cpu3 at mainbus0: apid 3 (application processor) cpu3: AMD Phenom(tm) II X6 1055T Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 2.81 GHz cpu3: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT cpu4 at mainbus0: apid 4 (application processor) cpu4: AMD Phenom(tm) II X6 1055T Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 2.81 GHz cpu4: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT cpu5 at mainbus0: apid 5 (application processor) cpu5: AMD Phenom(tm) II X6 1055T Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 2.81 GHz cpu5: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT ioapic0 at mainbus0: apid 6 pa 0xfec00000, version 21, 24 pins acpihpet0 at acpi0: 14318180 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 1 (P0P1) acpiprt2 at acpi0: bus 2 (PCE2) acpiprt3 at acpi0: bus 3 (PCE3) acpiprt4 at acpi0: bus -1 (PCE4) acpiprt5 at acpi0: bus 4 (PCE9) acpiprt6 at acpi0: bus 5 (PCEA) acpiprt7 at acpi0: bus 6 (P0PC) acpiprt8 at acpi0: bus 7 (PE20) acpiprt9 at acpi0: bus -1 (PE21) acpiprt10 at acpi0: bus -1 (PE22) acpiprt11 at acpi0: bus -1 (PE23) acpiec0 at acpi0 acpicpu0 at acpi0: C2, PSS acpicpu1 at acpi0: C2, PSS acpicpu2 at acpi0: C2, PSS acpicpu3 at acpi0: C2, PSS acpicpu4 at acpi0: C2, PSS acpicpu5 at acpi0: C2, PSS aibs0 at acpi0: GGRP GITM SITM acpibtn0 at acpi0: PWRB bios0: ROM list: 0xc0000/0xee00 0xcf000/0x4200! 0xd3800/0x8000 0xdb800/0x3000! pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "AMD RS780 Host" rev 0x00 ppb0 at pci0 dev 1 function 0 "AMD RS780 PCIE" rev 0x00 pci1 at ppb0 bus 1 vga1 at pci1 dev 5 function 0 vendor "ATI", unknown product 0x9714 rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ppb1 at pci0 dev 2 function 0 "AMD RS780 PCIE" rev 0x00: apic 6 int 18 (irq 10) pci2 at ppb1 bus 2 myx0 at pci2 dev 0 function 0 "Myricom Z8E" rev 0x00: apic 6 int 18 (irq 10), address 00:60:dd:47:8b:b5 ppb2 at pci0 dev 3 function 0 vendor "AMD", unknown product 0x960b rev 0x00: apic 6 int 19 (irq 11) pci3 at ppb2 bus 3 ix0 at pci3 dev 0 function 0 "Intel 10GbE DA Dual (82598)" rev 0x01: apic 6 int 19 (irq 11), address 00:1b:21:53:31:11 ix1 at pci3 dev 0 function 1 "Intel 10GbE DA Dual (82598)" rev 0x01: apic 6 int 16 (irq 10), address 00:1b:21:53:31:10 ppb3 at pci0 dev 9 function 0 "AMD RS780 PCIE" rev 0x00: apic 6 int 17 (irq 7) pci4 at ppb3 bus 4 "NEC PCIE-XHCI" rev 0x03 at pci4 dev 0 function 0 not configured ppb4 at pci0 dev 10 function 0 "AMD RS780 PCIE" rev 0x00: apic 6 int 18 (irq 10) pci5 at ppb4 bus 5 jmb0 at pci5 dev 0 function 0 "JMicron JMB361 IDE/SATA" rev 0x02 ahci0 at jmb0: apic 6 int 18 (irq 10), AHCI 1.0 ahci0: device on port 1 didn't come ready, TFD: 0xff<BSY,DRQ,ERR> ahci0: stopping the port, softreset slot 0 was still active. ahci0: unable to communicate with device on port 1 scsibus0 at ahci0: 32 targets pciide0 at jmb0: DMA, channel 0 wired to native-PCI, channel 1 wired to native-PCI pciide0: using apic 6 int 18 (irq 10) for native-PCI interrupt pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives) ahci1 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x40: apic 6 int 19 (irq 11), AHCI 1.2 scsibus1 at ahci1: 32 targets sd0 at scsibus1 targ 4 lun 0: <ATA, FB080C4080, > SCSI3 0/direct fixed sd0: 76319MB, 512 bytes/sec, 156301488 sec total sd1 at scsibus1 targ 5 lun 0: <ATA, FB080C4080, > SCSI3 0/direct fixed sd1: 76319MB, 512 bytes/sec, 156301488 sec total ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 6 int 18 (irq 10), version 1.0, legacy support ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 6 int 17 (irq 7) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "ATI EHCI root hub" rev 2.00/1.00 addr 1 ohci1 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 6 int 18 (irq 10), version 1.0, legacy support ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 6 int 17 (irq 7) usb1 at ehci1: USB revision 2.0 uhub1 at usb1 "ATI EHCI root hub" rev 2.00/1.00 addr 1 piixpm0 at pci0 dev 20 function 0 "ATI SBx00 SMBus" rev 0x41: SMBus disabled azalia0 at pci0 dev 20 function 2 "ATI SBx00 HD Audio" rev 0x40: apic 6 int 16 (irq 10) azalia0: codecs: Realtek/0x0892 audio0 at azalia0 pcib0 at pci0 dev 20 function 3 "ATI SB700 ISA" rev 0x40 ppb5 at pci0 dev 20 function 4 "ATI SB600 PCI" rev 0x40 pci6 at ppb5 bus 6 "VIA VT6306 FireWire" rev 0xc0 at pci6 dev 7 function 0 not configured ohci2 at pci0 dev 20 function 5 "ATI SB700 USB" rev 0x00: apic 6 int 18 (irq 10), version 1.0, legacy support ppb6 at pci0 dev 21 function 0 vendor "ATI", unknown product 0x43a0 rev 0x00: apic 6 int 16 (irq 255) pci7 at ppb6 bus 7 re0 at pci7 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E (0x2c00), apic 6 int 16 (irq 10), address 20:cf:30:0e:6e:f9 rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 4 ohci3 at pci0 dev 22 function 0 "ATI SB700 USB" rev 0x00: apic 6 int 18 (irq 10), version 1.0, legacy support ehci2 at pci0 dev 22 function 2 "ATI SB700 USB2" rev 0x00: apic 6 int 17 (irq 7) usb2 at ehci2: USB revision 2.0 uhub2 at usb2 "ATI EHCI root hub" rev 2.00/1.00 addr 1 pchb1 at pci0 dev 24 function 0 "AMD AMD64 10h HyperTransport" rev 0x00 pchb2 at pci0 dev 24 function 1 "AMD AMD64 10h Address Map" rev 0x00 pchb3 at pci0 dev 24 function 2 "AMD AMD64 10h DRAM Cfg" rev 0x00 km0 at pci0 dev 24 function 3 "AMD AMD64 10h Misc Cfg" rev 0x00 pchb4 at pci0 dev 24 function 4 "AMD AMD64 10h Link Cfg" rev 0x00 usb3 at ohci0: USB revision 1.0 uhub3 at usb3 "ATI OHCI root hub" rev 1.00/1.00 addr 1 usb4 at ohci1: USB revision 1.0 uhub4 at usb4 "ATI OHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 usb5 at ohci2: USB revision 1.0 uhub5 at usb5 "ATI OHCI root hub" rev 1.00/1.00 addr 1 usb6 at ohci3: USB revision 1.0 uhub6 at usb6 "ATI OHCI root hub" rev 1.00/1.00 addr 1 mtrr: Pentium Pro MTRR support vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root root on sd1a swap on sd1b dump on sd1b
