Re: igb requests for mbufs denied
On 2014-9-1, at 21:09, John-Mark Gurney j...@funkthat.com wrote: Still waiting for the other info I requested in my email... Sorry, which other info? (The message is not in dmesg, it's from netstat -m.) Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: igb requests for mbufs denied
On 2014-8-30, at 0:32, John-Mark Gurney j...@funkthat.com wrote: Also, what does sysctl dev.em and sysctl dev.igb show? The box has no em interfaces. [root@laurel: ~] sysctl dev.igb dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.RP03.LAN1 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1533 subvendor=0x1734 subdevice=0x11f1 class=0x02 dev.igb.0.%parent: pci4 dev.igb.0.nvm: -1 dev.igb.0.enable_aim: 1 dev.igb.0.fc: 3 dev.igb.0.rx_processing_limit: 100 dev.igb.0.dmac: 0 dev.igb.0.eee_disabled: 0 dev.igb.0.link_irq: 0 dev.igb.0.dropped: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.rx_overruns: 0 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.device_control: 135266881 dev.igb.0.rx_control: 4194304 dev.igb.0.interrupt_mask: 0 dev.igb.0.extended_int_mask: 2147483648 dev.igb.0.tx_buf_alloc: 0 dev.igb.0.rx_buf_alloc: 0 dev.igb.0.fc_high_water: 31328 dev.igb.0.fc_low_water: 31312 dev.igb.0.queue0.no_desc_avail: 0 dev.igb.0.queue0.tx_packets: 0 dev.igb.0.queue0.rx_packets: 0 dev.igb.0.queue0.rx_bytes: 0 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.missed_packets: 0 dev.igb.0.mac_stats.recv_no_buff: 0 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.xon_recvd: 0 dev.igb.0.mac_stats.xon_txd: 0 dev.igb.0.mac_stats.xoff_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 0 dev.igb.0.mac_stats.total_pkts_recvd: 0 dev.igb.0.mac_stats.good_pkts_recvd: 0 dev.igb.0.mac_stats.bcast_pkts_recvd: 0 dev.igb.0.mac_stats.mcast_pkts_recvd: 0 dev.igb.0.mac_stats.rx_frames_64: 0 dev.igb.0.mac_stats.rx_frames_65_127: 0 dev.igb.0.mac_stats.rx_frames_128_255: 0 dev.igb.0.mac_stats.rx_frames_256_511: 0 dev.igb.0.mac_stats.rx_frames_512_1023: 0 dev.igb.0.mac_stats.rx_frames_1024_1522: 0 dev.igb.0.mac_stats.good_octets_recvd: 0 dev.igb.0.mac_stats.good_octets_txd: 0 dev.igb.0.mac_stats.total_pkts_txd: 0 dev.igb.0.mac_stats.good_pkts_txd: 0 dev.igb.0.mac_stats.bcast_pkts_txd: 0 dev.igb.0.mac_stats.mcast_pkts_txd: 0 dev.igb.0.mac_stats.tx_frames_64: 0 dev.igb.0.mac_stats.tx_frames_65_127: 0 dev.igb.0.mac_stats.tx_frames_128_255: 0 dev.igb.0.mac_stats.tx_frames_256_511: 0 dev.igb.0.mac_stats.tx_frames_512_1023: 0 dev.igb.0.mac_stats.tx_frames_1024_1522: 0 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.interrupts.asserts: 0 dev.igb.0.interrupts.rx_pkt_timer: 0 dev.igb.0.interrupts.rx_abs_timer: 0 dev.igb.0.interrupts.tx_pkt_timer: 0 dev.igb.0.interrupts.tx_abs_timer: 0 dev.igb.0.interrupts.tx_queue_empty: 0 dev.igb.0.interrupts.tx_queue_min_thresh: 0 dev.igb.0.interrupts.rx_desc_min_thresh: 0 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.rx_pkt: 0 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 0 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.rx_good_bytes: 0 dev.igb.0.host.tx_good_bytes: 0 dev.igb.0.host.length_errors: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.header_redir_missed: 0 dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=0 handle=\_SB_.PCI0.RP04.LAN2 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1533 subvendor=0x1734 subdevice=0x11f1 class=0x02 dev.igb.1.%parent: pci5 dev.igb.1.nvm: -1 dev.igb.1.enable_aim: 1 dev.igb.1.fc: 3 dev.igb.1.rx_processing_limit: 100 dev.igb.1.dmac: 0 dev.igb.1.eee_disabled: 0 dev.igb.1.link_irq: 0 dev.igb.1.dropped: 0 dev.igb.1.tx_dma_fail: 0 dev.igb.1.rx_overruns: 0 dev.igb.1.watchdog_timeouts: 0 dev.igb.1.device_control: 135266881 dev.igb.1.rx_control: 4194304 dev.igb.1.interrupt_mask: 0 dev.igb.1.extended_int_mask: 2147483648 dev.igb.1.tx_buf_alloc: 0 dev.igb.1.rx_buf_alloc: 0 dev.igb.1.fc_high_water: 31328 dev.igb.1.fc_low_water: 31312 dev.igb.1.queue0.no_desc_avail: 0 dev.igb.1.queue0.tx_packets: 0 dev.igb.1.queue0.rx_packets: 0 dev.igb.1.queue0.rx_bytes: 0 dev.igb.1.queue0.lro_queued: 0 dev.igb.1.queue0.lro_flushed: 0 dev.igb.1.mac_stats.excess_coll: 0 dev.igb.1.mac_stats.single_coll: 0 dev.igb.1.mac_stats.multiple_coll: 0 dev.igb.1.mac_stats.late_coll: 0 dev.igb.1.mac_stats.collision_count: 0 dev.igb.1.mac_stats.symbol_errors: 0 dev.igb.1.mac_stats.sequence_errors: 0 dev.igb.1.mac_stats.defer_count: 0 dev.igb.1.mac_stats.missed_packets: 0 dev.igb.1.mac_stats.recv_no_buff: 0
Re: igb requests for mbufs denied
On 2014-8-30, at 7:24, Adrian Chadd adr...@freebsd.org wrote: What's the output of vmstat -z ? [root@laurel: ~] vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 99, 1, 99, 0, 0 UMA Zones: 1152, 0, 99, 0, 99, 0, 0 UMA Slabs: 80, 0, 15827, 23, 56426, 0, 0 UMA RCntSlabs: 88, 0,2283, 12,2283, 0, 0 UMA Hash: 256, 0, 5, 10, 8, 0, 0 4 Bucket:32, 0, 40, 835, 97770, 0, 0 8 Bucket:64, 0, 168, 514, 95742, 0, 0 16 Bucket: 128, 0, 91, 715, 44919, 202, 0 32 Bucket: 256, 0,3084,1386, 32197, 106, 0 64 Bucket: 512, 0,1579,2805, 37281, 101, 0 128 Bucket:1024, 0,2043, 361, 69687,82929, 0 vmem btag: 56, 0, 48977, 12722, 346536, 869, 0 VM OBJECT: 256, 0, 154107, 348, 389148, 0, 0 RADIX NODE: 144, 0, 184412, 22624, 674520, 49, 0 MAP:240, 0, 3, 61, 3, 0, 0 KMAP ENTRY: 128, 0, 9, 394, 9, 0, 0 MAP ENTRY: 128, 0, 760,1131, 430304, 0, 0 VMSPACE:448, 0, 28, 179, 11739, 0, 0 fakepg: 104, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 360, 0, 360, 0, 0 16: 16, 0,4012, 506, 518238, 0, 0 32: 32, 0,3760, 615, 36818, 0, 0 64: 64, 0, 14097, 74625, 3633439, 0, 0 128:128, 0,8331, 125930, 1771231, 0, 0 256:256, 0,1027, 82298, 1877944, 0, 0 512:512, 0, 854, 95994, 345637, 0, 0 1024: 1024, 0, 84, 168, 93998, 0, 0 2048: 2048, 0, 631, 733, 22957, 0, 0 4096: 4096, 0, 501, 32, 14417, 0, 0 SLEEPQUEUE: 80, 0, 193, 458, 193, 0, 0 uint64 pcpu: 8, 0,1602, 62,1602, 0, 0 Files: 80, 0, 147, 603, 1158104, 0, 0 TURNSTILE: 136, 0, 193, 187, 193, 0, 0 rl_entry:40, 0, 86, 414, 86, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1208, 0, 53, 52, 11825, 0, 0 THREAD:1168, 0, 171, 21, 189, 0, 0 cpuset: 72, 0, 87, 463, 149, 0, 0 audit_record: 1248, 0, 0, 0, 0, 0, 0 mbuf_packet:256, 52127340,3069, 791, 4919006,3110, 0 mbuf: 256, 52127340, 1,2259, 5017862,1247, 0 mbuf_cluster: 2048, 8144896,3860, 682, 583821,3377, 0 mbuf_jumbo_page: 4096, 1018097, 0, 12, 150937, 5, 0 mbuf_jumbo_9k: 9216, 301658, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 169682, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 248, 0, 0, 11088,12456274, 0, 0 ttyinq: 160, 0, 210, 240, 705, 0, 0 ttyoutq:256, 0, 109, 206, 367, 0, 0 ata_request:336, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr:24, 0, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 VNODE: 472, 0, 308616, 152, 2156908, 0, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 BUF TRIE: 144, 0, 951, 104997, 271764, 0, 0 NAMEI: 1024, 0, 0, 68, 6635607, 0, 0 S VFS Cache:108, 0, 284884, 25741, 2028838, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache:328, 0, 33803, 805, 275904, 0, 0 LTS VFS Cache: 368, 0, 10, 80, 30, 0, 0 DIRHASH: 1024, 0,2828, 8,4099, 0, 0 NCLNODE:528, 0, 14, 28, 14, 0, 0 fuse_ticket:224, 0, 0,
Re: igb requests for mbufs denied
Eggert, Lars wrote this message on Mon, Sep 01, 2014 at 09:17 +: On 2014-8-30, at 0:32, John-Mark Gurney j...@funkthat.com wrote: Also, what does sysctl dev.em and sysctl dev.igb show? The box has no em interfaces. Hard to know w/o complete info, i.e. complete dmesg... Still waiting for the other info I requested in my email... -- John-Mark Gurney Voice: +1 415 225 5579 All that I will do, has been done, All that I have, has not. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
On 2014-8-28, at 21:07, Steven Hartland kill...@multiplay.co.uk wrote: When you say you've bumped mbclusters and mbufs, was that in /boot/loader.conf or /etc/sysctl.conf. If the latter then thats too late for driver init so try the former. loader.conf Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: igb requests for mbufs denied
Eggert, Lars wrote this message on Thu, Aug 28, 2014 at 08:31 +: no matter what value I bump kern.ipc.nmbclusters and kern.ipc.nmbufs to, I still get requests for mbufs denied with igb interfaces, and the occasional connection stall, even when dialing down hw.igb.num_queues=1: Can you post the exact error message? Is this from dmesg? or? [root@laurel: ~] netstat -m 3070/1355/4425 mbufs in use (current/cache/total) 3069/773/3842/8144896 mbuf clusters in use (current/cache/total/max) 3069/772 mbuf+clusters out of packet secondary zone in use (current/cache) 0/6/6/1018097 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/301658 9k jumbo clusters in use (current/cache/total/max) 0/0/0/169682 16k jumbo clusters in use (current/cache/total/max) 6905K/1908K/8814K bytes allocated to network (current/cache/total) 73/3831/3091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 6/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile The weird thing to me is that your number of in use and other related counts are close to my machine which only has two Intel gige (em) interfaces... The only difference is that I have no denied mbufs or clusters... The box in question has six igb interfaces, 2x 'I210 Gigabit Network Connection' and 4x '82580 Gigabit Network Connection' and is running: FreeBSD laurel.muccbc.hq.netapp.com 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 Anyone have any ideas what else to try? Also, what does sysctl dev.em and sysctl dev.igb show? -- John-Mark Gurney Voice: +1 415 225 5579 All that I will do, has been done, All that I have, has not. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
What's the output of vmstat -z ? -a ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
On 8/28/2014 4:31 AM, Eggert, Lars wrote: Hi, no matter what value I bump kern.ipc.nmbclusters and kern.ipc.nmbufs to, I still get requests for mbufs denied with igb interfaces, and the occasional connection stall, even when dialing down hw.igb.num_queues=1: The box in question has six igb interfaces, 2x 'I210 Gigabit Network Connection' and 4x '82580 Gigabit Network Connection' and is running: FreeBSD laurel.muccbc.hq.netapp.com 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 Anyone have any ideas what else to try? A stab in the dark, but I ran into a bug where setting queues to 1 did not have the same positive impact as disabling msix in loader.conf hw.igb.enable_msix=0 ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
On 28.08.2014 12:31, Eggert, Lars wrote: Hi, no matter what value I bump kern.ipc.nmbclusters and kern.ipc.nmbufs to, I still get requests for mbufs denied with igb interfaces, and the occasional connection stall, even when dialing down hw.igb.num_queues=1: [root@laurel: ~] netstat -m 3070/1355/4425 mbufs in use (current/cache/total) 3069/773/3842/8144896 mbuf clusters in use (current/cache/total/max) 3069/772 mbuf+clusters out of packet secondary zone in use (current/cache) 0/6/6/1018097 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/301658 9k jumbo clusters in use (current/cache/total/max) 0/0/0/169682 16k jumbo clusters in use (current/cache/total/max) 6905K/1908K/8814K bytes allocated to network (current/cache/total) 73/3831/3091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 6/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile The box in question has six igb interfaces, 2x 'I210 Gigabit Network Connection' and 4x '82580 Gigabit Network Connection' and is running: FreeBSD laurel.muccbc.hq.netapp.com 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 Anyone have any ideas what else to try? Do you have jumbo frames turned on? Thanks, Lars ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
On 2014-8-28, at 13:17, Alexander V. Chernikov melif...@freebsd.org wrote: Do you have jumbo frames turned on? Nope. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: igb requests for mbufs denied
On 2014-8-28, at 13:02, Mike Tancsa m...@sentex.net wrote: hw.igb.enable_msix=0 Doesn't change things either, unfortunately. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: igb requests for mbufs denied
On 2014-8-28, at 10:31, Eggert, Lars l...@netapp.com wrote: 73/3831/3091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) I just noticed that these are already there just after boot, and then also don't seem to be increasing anymore (or only very slowly.) Just in case that gives someone an idea. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: igb requests for mbufs denied
Hi, also just noticed that there is a version 2.4.2 driver on Intel's site for these cards (https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=15815) whereas FreeBSD (incl. -CURRENT) is at 2.4.0. No changelog, unfortunately. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: igb requests for mbufs denied
On 8/28/2014 7:45 AM, Eggert, Lars wrote: Hi, also just noticed that there is a version 2.4.2 driver on Intel's site for these cards (https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=15815) whereas FreeBSD (incl. -CURRENT) is at 2.4.0. No changelog, unfortunately. There is a newer version in RELENG_10 and the change log just says newer IDs were added to support newer cards. There are some other minor changes which seems to have broken one of my 82574L cards however. ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
On Thu, Aug 28, 2014 at 4:27 AM, Eggert, Lars l...@netapp.com wrote: On 2014-8-28, at 10:31, Eggert, Lars l...@netapp.com wrote: 73/3831/3091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) I just noticed that these are already there just after boot, and then also don't seem to be increasing anymore (or only very slowly.) Just in case that gives someone an idea. Very weird. I guess a hackish way is to try removing cards or disabling interfaces one by one and see how it changes the behavior on boot? cheers, Hiren ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: igb requests for mbufs denied
When you say you've bumped mbclusters and mbufs, was that in /boot/loader.conf or /etc/sysctl.conf. If the latter then thats too late for driver init so try the former. - Original Message - From: Eggert, Lars l...@netapp.com Hi, no matter what value I bump kern.ipc.nmbclusters and kern.ipc.nmbufs to, I still get requests for mbufs denied with igb interfaces, and the occasional connection stall, even when dialing down hw.igb.num_queues=1: ... ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
