Re: problems with em(4) since update to driver 7.2.2
Hi, On Thu, May 5, 2011 at 1:20 AM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Wed, May 4, 2011 at 5:38 PM, Jack Vogel jfvo...@gmail.com wrote: I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. Actually, it can be trivially reproduced by tainting `error'. As it is uninitialized in HEAD, it's value can be _anything_, so let's mark it as explicitly invalid. diff -u ./if_em.c /data/src/freebsd/em-7.2.2/src/if_em.c --- ./if_em.c 2011-02-18 01:18:23.0 -0500 +++ /data/src/freebsd/em-7.2.2/src/if_em.c 2011-05-05 01:12:01.0 -0400 @@ -3912,7 +3912,7 @@ struct adapter *adapter = rxr-adapter; struct em_buffer *rxbuf; bus_dma_segment_t seg[1]; - int i, j, nsegs, error; + int i, j, nsegs, error = -1; The error pointed out in this thread pops up in the next boot. I put a call to kdb_enter() at the beginning of the function, helped with some textdump I got all the backtrace [0] for all the time em_setup_receive_ring() is called. All are exactly the same: kdb_enter_why(0,c09f6511,f391aaa8,c09be1e2,c09f6511,...) at kdb_enter_why+0x3b kdb_enter(c09f6511,0,3810,,5dc,...) at kdb_enter+0x19 em_setup_receive_ring(c3c8d600,c3c8d7a4,c3c96004,31fa,c3c8d600,...) at em_setup_receive_ring+0x22 em_setup_receive_structures(c3c96000,f15f2000,38,8100,3,...) at em_setup_receive_structures+0x26 em_init_locked(c3c96000,0,c09f5de5,414,1,...) at em_init_locked+0x2f2 em_ioctl(c3c7d000,80206934,c3ce9d00,c07b7a0b,c3f2a230,...) at em_ioctl+0x1c3 ifhwioctl(c3f2a230,f391ac34,c07b7a0b,c3f3e3d0,c08df1c0,...) at ifhwioctl+0x4b8 ifioctl(c3f3e3d0,80206934,c3ce9d00,c3f2a230,c3f2a230,...) at ifioctl+0x82 kern_ioctl(c3f2a230,3,80206934,c3ce9d00,c3ce9d00,...) at kern_ioctl+0xa8 ioctl(c3f2a230,f391acf8,c,c,f391ad2c,...) at ioctl+0xc5 syscall(f391ad38) at syscall+0x17d Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x4816ee23, esp = 0xbfbfe67c, ebp = 0xbfbfe698 --- This fully explain why the main loop in em_setup_receive_ring() is never entered, as we always verify `j == rxr-next_to_check' (provided that mbuf have been refreshed if some packet were transfered) and return the value on the stack. As of now, beside changing the call-site of em_setup_receive_ring() to ensure it is never re-entered, I'd guess that the patch I sent earlier today, is the only way to ensure that no junk is returned. I'd guess that the driver _is_ able to transmit, if the code was not explicitly calling em_stop() upon em_setup_receive_structures() failure. - Arnaud [0]: I wish that would have been as easy as in Linux, where a WARN() call do all the job automatically, but still, I should not hope for that much unless I am the one implementing it ... yes, free whining, it's 2a.m. ... - Arnaud The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. If you like Olivier I can make a version of em for you that also reverts the setup code the way I did for igb, see if that fixes it for you? Thanks for your patience, Jack ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
OK, but what this does not explain is why I do not see this if its so easily reproduced, what causes the failure case, any idea? As I said, given the code was not feasible for igb anyway I would not be unhappy about returning to the old way of doing things. Jack On Wed, May 4, 2011 at 11:03 PM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Thu, May 5, 2011 at 1:20 AM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Wed, May 4, 2011 at 5:38 PM, Jack Vogel jfvo...@gmail.com wrote: I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. Actually, it can be trivially reproduced by tainting `error'. As it is uninitialized in HEAD, it's value can be _anything_, so let's mark it as explicitly invalid. diff -u ./if_em.c /data/src/freebsd/em-7.2.2/src/if_em.c --- ./if_em.c 2011-02-18 01:18:23.0 -0500 +++ /data/src/freebsd/em-7.2.2/src/if_em.c 2011-05-05 01:12:01.0 -0400 @@ -3912,7 +3912,7 @@ struct adapter *adapter = rxr-adapter; struct em_buffer*rxbuf; bus_dma_segment_t seg[1]; - int i, j, nsegs, error; + int i, j, nsegs, error = -1; The error pointed out in this thread pops up in the next boot. I put a call to kdb_enter() at the beginning of the function, helped with some textdump I got all the backtrace [0] for all the time em_setup_receive_ring() is called. All are exactly the same: kdb_enter_why(0,c09f6511,f391aaa8,c09be1e2,c09f6511,...) at kdb_enter_why+0x3b kdb_enter(c09f6511,0,3810,,5dc,...) at kdb_enter+0x19 em_setup_receive_ring(c3c8d600,c3c8d7a4,c3c96004,31fa,c3c8d600,...) at em_setup_receive_ring+0x22 em_setup_receive_structures(c3c96000,f15f2000,38,8100,3,...) at em_setup_receive_structures+0x26 em_init_locked(c3c96000,0,c09f5de5,414,1,...) at em_init_locked+0x2f2 em_ioctl(c3c7d000,80206934,c3ce9d00,c07b7a0b,c3f2a230,...) at em_ioctl+0x1c3 ifhwioctl(c3f2a230,f391ac34,c07b7a0b,c3f3e3d0,c08df1c0,...) at ifhwioctl+0x4b8 ifioctl(c3f3e3d0,80206934,c3ce9d00,c3f2a230,c3f2a230,...) at ifioctl+0x82 kern_ioctl(c3f2a230,3,80206934,c3ce9d00,c3ce9d00,...) at kern_ioctl+0xa8 ioctl(c3f2a230,f391acf8,c,c,f391ad2c,...) at ioctl+0xc5 syscall(f391ad38) at syscall+0x17d Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x4816ee23, esp = 0xbfbfe67c, ebp = 0xbfbfe698 --- This fully explain why the main loop in em_setup_receive_ring() is never entered, as we always verify `j == rxr-next_to_check' (provided that mbuf have been refreshed if some packet were transfered) and return the value on the stack. As of now, beside changing the call-site of em_setup_receive_ring() to ensure it is never re-entered, I'd guess that the patch I sent earlier today, is the only way to ensure that no junk is returned. I'd guess that the driver _is_ able to transmit, if the code was not explicitly calling em_stop() upon em_setup_receive_structures() failure. - Arnaud [0]: I wish that would have been as easy as in Linux, where a WARN() call do all the job automatically, but still, I should not hope for that much unless I am the one implementing it ... yes, free whining, it's 2a.m. ... - Arnaud The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. If you like Olivier I can make a version of em for you that also reverts the setup code the way I did for igb, see if that fixes it for you? Thanks for your patience, Jack ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hello, 2011/5/4 Arnaud Lacombe lacom...@gmail.com: Hi, On Wed, May 4, 2011 at 3:58 AM, Olivier Smedts oliv...@gid0.org wrote: em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: Could not setup receive structures em0: Could not setup receive structures What can we do to help you debug this ? At some point in time, in late February, I had the same issue on a 6-interface machine. I tracked this down to the fact that the main loop in em_setup_receive_ring() was not being entered. This resulted in junk being returned as `error' is not explicitly initialized. At the time, the following patch worked for me. Without it the driver was unable to initialize with RX/TX ring's size of 512. With it, ring's size of 1024 initialized fine. diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c index fb6ed67..f02059a 100644 --- a/sys/dev/e1000/if_em.c +++ b/sys/dev/e1000/if_em.c @@ -3901,7 +3901,7 @@ em_setup_receive_ring(struct rx_ring *rxr) struct adapter *adapter = rxr-adapter; struct em_buffer *rxbuf; bus_dma_segment_t seg[1]; - int i, j, nsegs, error; + int i, j, nsegs, error = 0; This patch made the trick for me. I'll post what Jack asked for in the following mail. I did not dig much more at the time, but I was definitively seeing an odd behavior. Anyhow, I am no longer able to reproduce this with 7.2.3, so cannot dig in more details. Btw, I wish you all luck, it took me nearly two full months to convince Jack (and other FreeBSD devs) that there was a bug in the mbuf refresh code. - Arnaud -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hello, (sorry for dual posting) 2011/5/4 Jack Vogel jfvo...@gmail.com: I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. The computer is a HP Compaq 8100 Elite Convertible Minitower PC. Here is what I have with the new driver and Arnaud Lacombe's patch. %uname -a FreeBSD zozo.afpicl.lan 9.0-CURRENT FreeBSD 9.0-CURRENT #0 r219752:221420: Wed May 4 11:16:37 CEST 2011 r...@zozo.afpicl.lan:/usr/obj/usr/src/sys/CORE amd64 %pciconf -lv hostb0@pci0:0:0:0: class=0x06 card=0x304b103c chip=0xd1318086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI pcib1@pci0:0:3:0: class=0x060400 card=0x304b103c chip=0xd1388086 rev=0x11 hdr=0x01 vendor = 'Intel Corporation' class = bridge subclass = PCI-PCI none0@pci0:0:8:0: class=0x088000 card=0x004b003c chip=0xd1558086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = base peripheral none1@pci0:0:8:1: class=0x088000 card=0x004b003c chip=0xd1568086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = base peripheral none2@pci0:0:8:2: class=0x088000 card=0x004b003c chip=0xd1578086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = base peripheral none3@pci0:0:8:3: class=0x088000 card=0x004b003c chip=0xd1588086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = base peripheral none4@pci0:0:16:0: class=0x088000 card=0x004b003c chip=0xd1508086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = base peripheral none5@pci0:0:16:1: class=0x088000 card=0x004b003c chip=0xd1518086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' class = base peripheral none6@pci0:0:22:0: class=0x078000 card=0x304b103c chip=0x3b648086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' class = simple comms none7@pci0:0:22:3: class=0x070002 card=0x304b103c chip=0x3b678086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' class = simple comms subclass = UART em0@pci0:0:25:0:class=0x02 card=0x304b103c chip=0x10ef8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet ehci0@pci0:0:26:0: class=0x0c0320 card=0x304b103c chip=0x3b3c8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' class = serial bus subclass = USB hdac1@pci0:0:27:0: class=0x040300 card=0x304b103c chip=0x3b568086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' class = multimedia subclass = HDA pcib2@pci0:0:28:0: class=0x060400 card=0x304b103c chip=0x3b428086 rev=0x05 hdr=0x01 vendor = 'Intel Corporation' class = bridge subclass = PCI-PCI pcib3@pci0:0:28:4: class=0x060400 card=0x304b103c chip=0x3b4a8086 rev=0x05 hdr=0x01 vendor = 'Intel Corporation' class = bridge subclass = PCI-PCI pcib4@pci0:0:28:6: class=0x060400 card=0x304b103c chip=0x3b4e8086 rev=0x05 hdr=0x01 vendor = 'Intel Corporation' class = bridge subclass = PCI-PCI ehci1@pci0:0:29:0: class=0x0c0320 card=0x304b103c chip=0x3b348086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' class = serial bus subclass = USB pcib5@pci0:0:30:0: class=0x060401 card=0x304b103c chip=0x244e8086 rev=0xa5 hdr=0x01 vendor = 'Intel Corporation' device = '82801 Family (ICH2/3/4/5/6/7/8/9,63xxESB) Hub Interface to PCI Bridge' class = bridge subclass = PCI-PCI isab0@pci0:0:31:0: class=0x060100 card=0x304b103c chip=0x3b0a8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x304b103c chip=0x3b228086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' device = 'IBEX AHCI Controller(6Port) (Intel Q57 Express)' class = mass storage subclass = SATA vgapci0@pci0:1:0:0: class=0x03 card=0x10021002 chip=0x94981002 rev=0x00 hdr=0x00
Re: problems with em(4) since update to driver 7.2.2
Hi, On Thu, May 5, 2011 at 2:59 AM, Jack Vogel jfvo...@gmail.com wrote: OK, but what this does not explain is why I do not see this if its so easily reproduced, what causes the failure case, any idea? It is completely random as it depends on the content of the stack. I spent 3 or 4 hours trying to reproduce it using different approach on different platform, with different version of the code and failed. And once `error' was explicitly colored, it popped up. That's the beauty of error related with uninitialized variable. - Arnaud As I said, given the code was not feasible for igb anyway I would not be unhappy about returning to the old way of doing things. I am not sure what you mean by old way of doing thing, but I'd guess that the ring only need to be setup on a few occasion, like initialization and MTU transition. I'm not sure either how other driver manage their ring. Jack On Wed, May 4, 2011 at 11:03 PM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Thu, May 5, 2011 at 1:20 AM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Wed, May 4, 2011 at 5:38 PM, Jack Vogel jfvo...@gmail.com wrote: I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. Actually, it can be trivially reproduced by tainting `error'. As it is uninitialized in HEAD, it's value can be _anything_, so let's mark it as explicitly invalid. diff -u ./if_em.c /data/src/freebsd/em-7.2.2/src/if_em.c --- ./if_em.c 2011-02-18 01:18:23.0 -0500 +++ /data/src/freebsd/em-7.2.2/src/if_em.c 2011-05-05 01:12:01.0 -0400 @@ -3912,7 +3912,7 @@ struct adapter *adapter = rxr-adapter; struct em_buffer *rxbuf; bus_dma_segment_t seg[1]; - int i, j, nsegs, error; + int i, j, nsegs, error = -1; The error pointed out in this thread pops up in the next boot. I put a call to kdb_enter() at the beginning of the function, helped with some textdump I got all the backtrace [0] for all the time em_setup_receive_ring() is called. All are exactly the same: kdb_enter_why(0,c09f6511,f391aaa8,c09be1e2,c09f6511,...) at kdb_enter_why+0x3b kdb_enter(c09f6511,0,3810,,5dc,...) at kdb_enter+0x19 em_setup_receive_ring(c3c8d600,c3c8d7a4,c3c96004,31fa,c3c8d600,...) at em_setup_receive_ring+0x22 em_setup_receive_structures(c3c96000,f15f2000,38,8100,3,...) at em_setup_receive_structures+0x26 em_init_locked(c3c96000,0,c09f5de5,414,1,...) at em_init_locked+0x2f2 em_ioctl(c3c7d000,80206934,c3ce9d00,c07b7a0b,c3f2a230,...) at em_ioctl+0x1c3 ifhwioctl(c3f2a230,f391ac34,c07b7a0b,c3f3e3d0,c08df1c0,...) at ifhwioctl+0x4b8 ifioctl(c3f3e3d0,80206934,c3ce9d00,c3f2a230,c3f2a230,...) at ifioctl+0x82 kern_ioctl(c3f2a230,3,80206934,c3ce9d00,c3ce9d00,...) at kern_ioctl+0xa8 ioctl(c3f2a230,f391acf8,c,c,f391ad2c,...) at ioctl+0xc5 syscall(f391ad38) at syscall+0x17d Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x4816ee23, esp = 0xbfbfe67c, ebp = 0xbfbfe698 --- This fully explain why the main loop in em_setup_receive_ring() is never entered, as we always verify `j == rxr-next_to_check' (provided that mbuf have been refreshed if some packet were transfered) and return the value on the stack. As of now, beside changing the call-site of em_setup_receive_ring() to ensure it is never re-entered, I'd guess that the patch I sent earlier today, is the only way to ensure that no junk is returned. I'd guess that the driver _is_ able to transmit, if the code was not explicitly calling em_stop() upon em_setup_receive_structures() failure. - Arnaud [0]: I wish that would have been as easy as in Linux, where a WARN() call do all the job automatically, but still, I should not hope for that much unless I am the one implementing it ... yes, free whining, it's 2a.m. ... - Arnaud The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. If you like Olivier I can make a version of em for you that also reverts the
Re: problems with em(4) since update to driver 7.2.2
Hi, On Wed, May 4, 2011 at 3:00 AM, Alastair Hogge a...@fastmail.fm wrote: [.] I also tried 2x, 4x 25600 for max mbuff clusters via kern.ipc.nmbclusters. This didn't help. For the record, I did the math yestarday, checked the code. By default, a machine with 6 82574L-backed em(4) interfaces, with only 3 used (ie. brought up), initializes and work just fine with as low as 3076 mbuf clusters (1024*3 + 2). It has been transferring about 28k pps or 20Mbps of traffic (ICMP ping flood) since for the last 10h. Here is the `netstat -m' output: # netstat -m 2879/916/3795 mbufs in use (current/cache/total) 2877/199/3076/3076 mbuf clusters in use (current/cache/total/max) 2877/199 mbuf+clusters out of packet secondary zone in use (current/cache) 0/2/2/1537 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/768 9k jumbo clusters in use (current/cache/total/max) 0/0/0/384 16k jumbo clusters in use (current/cache/total/max) 6473K/635K/7108K bytes allocated to network (current/cache/total) 0/540580029/268859859 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/5/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines and, yes, allocation denial has sky-rocketed, but beside that the driver is stable. In that case, the uninitialized issue did not happen when the system booted. The complete machine should be able to initialize properly with 6146 clusters. - Arnaud ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
On Thu, May 5, 2011 at 7:21 AM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Thu, May 5, 2011 at 2:59 AM, Jack Vogel jfvo...@gmail.com wrote: OK, but what this does not explain is why I do not see this if its so easily reproduced, what causes the failure case, any idea? It is completely random as it depends on the content of the stack. I spent 3 or 4 hours trying to reproduce it using different approach on different platform, with different version of the code and failed. And once `error' was explicitly colored, it popped up. That's the beauty of error related with uninitialized variable. - Arnaud As I said, given the code was not feasible for igb anyway I would not be unhappy about returning to the old way of doing things. I am not sure what you mean by old way of doing thing, but I'd guess that the ring only need to be setup on a few occasion, like initialization and MTU transition. I'm not sure either how other driver manage their ring. The old way was as the code is in igb now, on each entry to this setup it would completely wipe the descriptor memory, then release all mbufs, and initialize from scratch. Its only because of this lazy reinit, meaning only the range from next_to_refresh to next_to_check is reset, that this problem can happen. For igb the reason this will not work, is it requires you to set E1000_RDH(i) to next_to_check, and in fact, the hardware prohibits the write, its ALWAYS 0 after a reset. The reason for this is that the hardware wishes to manage the head index and not software. Anyway, I see the problematic code path, its only when you skip the while loop altogether. I'm surprised the compiler did not complain about this, its usually so anal. Jack ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/5/5 Jack Vogel jfvo...@gmail.com: Anyway, I see the problematic code path, its only when you skip the while loop altogether. I'm surprised the compiler did not complain about this, its usually so anal. Could it be related to the compiler (clang) or some optimization flags ? -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Not sure, I wondered if those seeing this had some special sequence of actions they took for granted that is different than what we do in house... In any case, the init really is ultimately a correctness thing, so let's just call it good :) Jack On Thu, May 5, 2011 at 11:16 AM, Olivier Smedts oliv...@gid0.org wrote: 2011/5/5 Jack Vogel jfvo...@gmail.com: Anyway, I see the problematic code path, its only when you skip the while loop altogether. I'm surprised the compiler did not complain about this, its usually so anal. Could it be related to the compiler (clang) or some optimization flags ? -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org- against HTML email vCards X www: http://www.gid0.org- against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/5/4 Jack Vogel jfvo...@gmail.com: It has nothing to do with load, it has to do with the prerequisites to init your interfaces. The amount you need is fixed, it doesn't vary with load. Every RX descriptor needs one, so its simple math, number-of-interfaces X number-of-queues X size of the ring. If you have other network interfaces beside Intel they also consume mbufs remember. Only one network interface. # kldunload if_em.ko (the old one) # sysctl kern.ipc.nmbclusters=512000 (I also tried with lower and more meaningful values) # kldload ./if_em.ko (the new one) # dmesg em0: detached pci0: network, ethernet at device 25.0 (no driver attached) em0: Intel(R) PRO/1000 Network Connection 7.2.3 port 0x2100-0x211f mem 0xf000-0xf001,0xf0025000-0xf0025fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: Could not setup receive structures em0: Could not setup receive structures What can we do to help you debug this ? -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
On Tue, May 03, 2011 at 11:50:53PM +0200, Michael Schmiedgen wrote: On 03.05.2011 23:24, Jack Vogel wrote: If you get the setup receive structures fail, then increase the nmbclusters. If you use standard MTU then what you need are mbufs, and standard size clusters (2K). Only when you use jumbo frames will you need larger. You must configure enough, its that simple. I doubled the nmbclusters as well. But nothing happened. I have no load on this machine and nothing special configured. Thanks, Michael Just a me too. I've been following -CURRENT(r221415) but I keep /sys/dev/e1000 at r218909 to keep em0 working without problems on -CURRENT. I also tried 2x, 4x 25600 for max mbuff clusters via kern.ipc.nmbclusters. This didn't help. -alastair ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi, On Wed, May 4, 2011 at 3:58 AM, Olivier Smedts oliv...@gid0.org wrote: em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: Could not setup receive structures em0: Could not setup receive structures What can we do to help you debug this ? At some point in time, in late February, I had the same issue on a 6-interface machine. I tracked this down to the fact that the main loop in em_setup_receive_ring() was not being entered. This resulted in junk being returned as `error' is not explicitly initialized. At the time, the following patch worked for me. Without it the driver was unable to initialize with RX/TX ring's size of 512. With it, ring's size of 1024 initialized fine. diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c index fb6ed67..f02059a 100644 --- a/sys/dev/e1000/if_em.c +++ b/sys/dev/e1000/if_em.c @@ -3901,7 +3901,7 @@ em_setup_receive_ring(struct rx_ring *rxr) struct adapter *adapter = rxr-adapter; struct em_buffer*rxbuf; bus_dma_segment_t seg[1]; - int i, j, nsegs, error; + int i, j, nsegs, error = 0; I did not dig much more at the time, but I was definitively seeing an odd behavior. Anyhow, I am no longer able to reproduce this with 7.2.3, so cannot dig in more details. Btw, I wish you all luck, it took me nearly two full months to convince Jack (and other FreeBSD devs) that there was a bug in the mbuf refresh code. - Arnaud ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi, On Tue, May 3, 2011 at 7:25 PM, Olivier Smedts oliv...@gid0.org wrote: 2011/5/4 Arnaud Lacombe lacom...@gmail.com: A more rude version might be Why the frak my network adapter stopped working with the default setting ? :) ...on a -STABLE branch Maybe you should not have picked the rude version, Jack has a relatively low cut-off frequency :-) - Arnaud ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
No, I do not Arnaud. But I refuse to rise to rude and uncivil behavior. Its your behavior again and again which causes you to not get responses, not my willingness to help and respond to those that behave like respectful customers and adults. Jack On Wed, May 4, 2011 at 10:20 AM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Tue, May 3, 2011 at 7:25 PM, Olivier Smedts oliv...@gid0.org wrote: 2011/5/4 Arnaud Lacombe lacom...@gmail.com: A more rude version might be Why the frak my network adapter stopped working with the default setting ? :) ...on a -STABLE branch Maybe you should not have picked the rude version, Jack has a relatively low cut-off frequency :-) - Arnaud ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi, On Wed, May 4, 2011 at 1:24 PM, Jack Vogel jfvo...@gmail.com wrote: No, I do not Arnaud. But I refuse to rise to rude and uncivil behavior. Its your behavior again and again which causes you to not get responses, not my willingness to help and respond to those that behave like respectful customers and adults. Obviously, I am no longer the only one finding that em(4) has unacceptable issue, this thread is the proof. - Arrnaud Jack On Wed, May 4, 2011 at 10:20 AM, Arnaud Lacombe lacom...@gmail.com wrote: Hi, On Tue, May 3, 2011 at 7:25 PM, Olivier Smedts oliv...@gid0.org wrote: 2011/5/4 Arnaud Lacombe lacom...@gmail.com: A more rude version might be Why the frak my network adapter stopped working with the default setting ? :) ...on a -STABLE branch Maybe you should not have picked the rude version, Jack has a relatively low cut-off frequency :-) - Arnaud ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/5/4 Arnaud Lacombe lacom...@gmail.com: Obviously, I am no longer the only one finding that em(4) has unacceptable issue, this thread is the proof. Right, and Jack seems to be willing to help, he asked something (I'll reply tomorrow when I'll be in front of the hardware) and is trying to find the same hardware. Now please not get out of the thread, everybody back to work :) -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. If you like Olivier I can make a version of em for you that also reverts the setup code the way I did for igb, see if that fixes it for you? Thanks for your patience, Jack ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi, On Wed, May 4, 2011 at 5:38 PM, Jack Vogel jfvo...@gmail.com wrote: I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? That would be consistent with what I reported back in February. I'll try to see if I can have a look at that on our platform tonight. - Arnaud You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. If you like Olivier I can make a version of em for you that also reverts the setup code the way I did for igb, see if that fixes it for you? Thanks for your patience, Jack ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi, On Wed, May 4, 2011 at 5:38 PM, Jack Vogel jfvo...@gmail.com wrote: I have had my validation engineer busy all day, we have tried both a 9 kernel as well as 8.2, using the code from HEAD, and we cannot reproduce this problem. Actually, it can be trivially reproduced by tainting `error'. As it is uninitialized in HEAD, it's value can be _anything_, so let's mark it as explicitly invalid. diff -u ./if_em.c /data/src/freebsd/em-7.2.2/src/if_em.c --- ./if_em.c 2011-02-18 01:18:23.0 -0500 +++ /data/src/freebsd/em-7.2.2/src/if_em.c 2011-05-05 01:12:01.0 -0400 @@ -3912,7 +3912,7 @@ struct adapter *adapter = rxr-adapter; struct em_buffer*rxbuf; bus_dma_segment_t seg[1]; - int i, j, nsegs, error; + int i, j, nsegs, error = -1; The error pointed out in this thread pops up in the next boot. - Arnaud The data your netstat -m shows suggests to me that what's happening is somehow setup of the receive ring is running more than once maybe?? You asked at one point how this could go into STABLE, well, because not only here at Intel, but at lots of external customers this code has been used and tested thoroughly. I am not calling into question your problem, but until I understand what it is I cannot fix it :) The thing I am guessing right now is the culprit is the setup code, the reason is that when I ported to the igb driver I found that it did not work on our newer hardware, and so I went back to the older version of setup for igb. Now, even though I have not seen hardware fail with em, maybe there is some. To help me give me a complete pciconf -lv, and if its a namebrand system tell me that, including all hardware in it. If you like Olivier I can make a version of em for you that also reverts the setup code the way I did for igb, see if that fixes it for you? Thanks for your patience, Jack ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hello, 2011/4/27 Jack Vogel jfvo...@gmail.com: If you get cannot setup receive structures you cannot go on and try to use the thing :) It means you have inadequate mbuf clusters to setup your receive side, you simply have to increase it and reload the driver. I tried increasing kern.ipc.nmbjumbo* (is it what you suggested ?). Values doubled : kern.ipc.nmbjumbo16: 6400 kern.ipc.nmbjumbo9: 12800 kern.ipc.nmbjumbop: 25600 And unloaded / reloaded the kernel module. Still no luck, same problem, on latest 9-CURRENT (r221363). %sysctl -a | grep mbuf dev.em.0.mbuf_alloc_fail: 0 What can I do ? Do you want a dump of sysctl dev.em with old and new if_em module ? Thanks Here is what gives me netstat -m with my new 9-CURRENT kernel but with old (working, after some time of computer use) if_em.ko : 1027/3458/4485 mbufs in use (current/cache/total) 1024/2066/3090/25600 mbuf clusters in use (current/cache/total/max) 1024/1792 mbuf+clusters out of packet secondary zone in use (current/cache) 0/367/367/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2304K/6464K/8769K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines And here is the output with the new (non-working) if_em.ko : 1029/3456/4485 mbufs in use (current/cache/total) 1023/2067/3090/25600 mbuf clusters in use (current/cache/total/max) 1023/1793 mbuf+clusters out of packet secondary zone in use (current/cache) 0/367/367/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2303K/6466K/8769K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines I've got the em0: Could not setup receive structures messages with the new if_em.ko even in single user mode. No network connectivity. I tried removing all other network-related modules (vboxnet, ipfw...) and still have this problem (again, even when booting in single-user mode). My network card is em0@pci0:0:25:0: class=0x02 card=0x304b103c chip=0x10ef8086 rev=0x05 hdr=0x00. I'm using a stripped-down GENERIC amd64 kernel (no network, no scsi, no raid...), a nearly empty sysctl.conf and loader.conf (except module loading). I saw at the time of the commit that an MFC to 8-STABLE was planned, but I don't think it should happen so soon. Given that my network adapter was previously working well before the em driver update, can't this be considerd a serious regression ? Thanks, Olivier -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
If you get the setup receive structures fail, then increase the nmbclusters. If you use standard MTU then what you need are mbufs, and standard size clusters (2K). Only when you use jumbo frames will you need larger. You must configure enough, its that simple. Jack On Tue, May 3, 2011 at 2:13 PM, Michael Schmiedgen schmied...@gmx.netwrote: Hi, I have the very same problem. - GENERIC 9.0-CURRENT (April 28) - em0 PRO/1000 7.2.3 - em0: Could not setup receive structures On 03.05.2011 10:58, Olivier Smedts wrote: I tried increasing kern.ipc.nmbjumbo* (is it what you suggested ?). Values doubled : kern.ipc.nmbjumbo16: 6400 kern.ipc.nmbjumbo9: 12800 kern.ipc.nmbjumbop: 25600 And unloaded / reloaded the kernel module. Still no luck, same problem, on latest 9-CURRENT (r221363). Same here. If I should provide some more configuration settings, please let me know. Michael ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi, I have the very same problem. - GENERIC 9.0-CURRENT (April 28) - em0 PRO/1000 7.2.3 - em0: Could not setup receive structures On 03.05.2011 10:58, Olivier Smedts wrote: I tried increasing kern.ipc.nmbjumbo* (is it what you suggested ?). Values doubled : kern.ipc.nmbjumbo16: 6400 kern.ipc.nmbjumbo9: 12800 kern.ipc.nmbjumbop: 25600 And unloaded / reloaded the kernel module. Still no luck, same problem, on latest 9-CURRENT (r221363). Same here. If I should provide some more configuration settings, please let me know. Michael ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
On 03.05.2011 23:24, Jack Vogel wrote: If you get the setup receive structures fail, then increase the nmbclusters. If you use standard MTU then what you need are mbufs, and standard size clusters (2K). Only when you use jumbo frames will you need larger. You must configure enough, its that simple. I doubled the nmbclusters as well. But nothing happened. I have no load on this machine and nothing special configured. Thanks, Michael ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
It has nothing to do with load, it has to do with the prerequisites to init your interfaces. The amount you need is fixed, it doesn't vary with load. Every RX descriptor needs one, so its simple math, number-of-interfaces X number-of-queues X size of the ring. If you have other network interfaces beside Intel they also consume mbufs remember. Jack On Tue, May 3, 2011 at 2:50 PM, Michael Schmiedgen schmied...@gmx.netwrote: On 03.05.2011 23:24, Jack Vogel wrote: If you get the setup receive structures fail, then increase the nmbclusters. If you use standard MTU then what you need are mbufs, and standard size clusters (2K). Only when you use jumbo frames will you need larger. You must configure enough, its that simple. I doubled the nmbclusters as well. But nothing happened. I have no load on this machine and nothing special configured. Thanks, Michael ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/5/4 Arnaud Lacombe lacom...@gmail.com: A more rude version might be Why the frak my network adapter stopped working with the default setting ? :) ...on a -STABLE branch -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Hi Jack, On Tue, May 3, 2011 at 6:00 PM, Jack Vogel jfvo...@gmail.com wrote: It has nothing to do with load, it has to do with the prerequisites to init your interfaces. The amount you need is fixed, it doesn't vary with load. Every RX descriptor needs one, so its simple math, number-of-interfaces X number-of-queues X size of the ring. I guess the question is more: why would I need N*M (M 1) nmbclusters with driver version X when driver version X-1 worked perfectly fine, from my daily-average-user point of view, with N nmbclusters ? A more rude version might be Why the frak my network adapter stopped working with the default setting ? :) - Arnaud If you have other network interfaces beside Intel they also consume mbufs remember. Jack On Tue, May 3, 2011 at 2:50 PM, Michael Schmiedgen schmied...@gmx.netwrote: On 03.05.2011 23:24, Jack Vogel wrote: If you get the setup receive structures fail, then increase the nmbclusters. If you use standard MTU then what you need are mbufs, and standard size clusters (2K). Only when you use jumbo frames will you need larger. You must configure enough, its that simple. I doubled the nmbclusters as well. But nothing happened. I have no load on this machine and nothing special configured. Thanks, Michael ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/3/31 Jack Vogel jfvo...@gmail.com: This problem happens for only one reason, you have insufficient mbufs to fill your rx ring. Its odd that it would differ when its static versus a loadable module though! With the 7.2.2 driver you also will use different mbuf pools depending on the MTU you are using. If you use jumbo frames it will use 4K clusters, if you go to 9K jumbos it will use 9K mbuf clusters. The number of these allocated by default is small (like 6400 small :). I would use 'netstat -m' to see what the pools look like. Now that I think about it, the reason it might fail as loaded while not as built in is you get allocation of the mbufs first when static, and something else is taking them before you can load when loadable?? Sorry to be quite late on this, Here is what gives me netstat -m with my new 9-CURRENT kernel but with old (working, after some time of computer use) if_em.ko : 1027/3458/4485 mbufs in use (current/cache/total) 1024/2066/3090/25600 mbuf clusters in use (current/cache/total/max) 1024/1792 mbuf+clusters out of packet secondary zone in use (current/cache) 0/367/367/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2304K/6464K/8769K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines And here is the output with the new (non-working) if_em.ko : 1029/3456/4485 mbufs in use (current/cache/total) 1023/2067/3090/25600 mbuf clusters in use (current/cache/total/max) 1023/1793 mbuf+clusters out of packet secondary zone in use (current/cache) 0/367/367/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2303K/6466K/8769K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines I've got the em0: Could not setup receive structures messages with the new if_em.ko even in single user mode. No network connectivity. I tried removing all other network-related modules (vboxnet, ipfw...) and still have this problem (again, even when booting in single-user mode). My network card is em0@pci0:0:25:0:class=0x02 card=0x304b103c chip=0x10ef8086 rev=0x05 hdr=0x00. I'm using a stripped-down GENERIC amd64 kernel (no network, no scsi, no raid...), a nearly empty sysctl.conf and loader.conf (except module loading). I saw at the time of the commit that an MFC to 8-STABLE was planned, but I don't think it should happen so soon. Given that my network adapter was previously working well before the em driver update, can't this be considerd a serious regression ? Thanks, Olivier -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
If you get cannot setup receive structures you cannot go on and try to use the thing :) It means you have inadequate mbuf clusters to setup your receive side, you simply have to increase it and reload the driver. Jack On Wed, Apr 27, 2011 at 5:39 AM, Olivier Smedts oliv...@gid0.org wrote: 2011/3/31 Jack Vogel jfvo...@gmail.com: This problem happens for only one reason, you have insufficient mbufs to fill your rx ring. Its odd that it would differ when its static versus a loadable module though! With the 7.2.2 driver you also will use different mbuf pools depending on the MTU you are using. If you use jumbo frames it will use 4K clusters, if you go to 9K jumbos it will use 9K mbuf clusters. The number of these allocated by default is small (like 6400 small :). I would use 'netstat -m' to see what the pools look like. Now that I think about it, the reason it might fail as loaded while not as built in is you get allocation of the mbufs first when static, and something else is taking them before you can load when loadable?? Sorry to be quite late on this, Here is what gives me netstat -m with my new 9-CURRENT kernel but with old (working, after some time of computer use) if_em.ko : 1027/3458/4485 mbufs in use (current/cache/total) 1024/2066/3090/25600 mbuf clusters in use (current/cache/total/max) 1024/1792 mbuf+clusters out of packet secondary zone in use (current/cache) 0/367/367/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2304K/6464K/8769K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines And here is the output with the new (non-working) if_em.ko : 1029/3456/4485 mbufs in use (current/cache/total) 1023/2067/3090/25600 mbuf clusters in use (current/cache/total/max) 1023/1793 mbuf+clusters out of packet secondary zone in use (current/cache) 0/367/367/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2303K/6466K/8769K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines I've got the em0: Could not setup receive structures messages with the new if_em.ko even in single user mode. No network connectivity. I tried removing all other network-related modules (vboxnet, ipfw...) and still have this problem (again, even when booting in single-user mode). My network card is em0@pci0:0:25:0:class=0x02 card=0x304b103c chip=0x10ef8086 rev=0x05 hdr=0x00. I'm using a stripped-down GENERIC amd64 kernel (no network, no scsi, no raid...), a nearly empty sysctl.conf and loader.conf (except module loading). I saw at the time of the commit that an MFC to 8-STABLE was planned, but I don't think it should happen so soon. Given that my network adapter was previously working well before the em driver update, can't this be considerd a serious regression ? Thanks, Olivier -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org- against HTML email vCards X www: http://www.gid0.org- against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/4/27 Jack Vogel jfvo...@gmail.com: If you get cannot setup receive structures you cannot go on and try to use the thing :) It means you have inadequate mbuf clusters to setup your receive side, you simply have to increase it and reload the driver. Thanks for your answer. I'll try and let you know if this works, but why don't have I enough mbuf clusters ? Isn't this driver supposed to just work out of the box on a nearly GENERIC (but with network devices as modules) kernel ? -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
On 4/27/2011 2:35 PM, Olivier Smedts wrote: 2011/4/27 Jack Vogel jfvo...@gmail.com: If you get cannot setup receive structures you cannot go on and try to use the thing :) It means you have inadequate mbuf clusters to setup your receive side, you simply have to increase it and reload the driver. Thanks for your answer. I'll try and let you know if this works, but why don't have I enough mbuf clusters ? Isn't this driver supposed to just work out of the box on a nearly GENERIC (but with network devices as modules) kernel ? Are you testing with what is in HEAD ? ie. 7.2.3 or something else ? Your subject line implies something else. ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
2011/4/27 Mike Tancsa m...@sentex.net: On 4/27/2011 2:35 PM, Olivier Smedts wrote: 2011/4/27 Jack Vogel jfvo...@gmail.com: If you get cannot setup receive structures you cannot go on and try to use the thing :) It means you have inadequate mbuf clusters to setup your receive side, you simply have to increase it and reload the driver. Thanks for your answer. I'll try and let you know if this works, but why don't have I enough mbuf clusters ? Isn't this driver supposed to just work out of the box on a nearly GENERIC (but with network devices as modules) kernel ? Are you testing with what is in HEAD ? ie. 7.2.3 or something else ? Your subject line implies something else. I'm using what's in HEAD since r219753, the commit which updated the em driver from version 7.1.9 to 7.2.2 and broke it at least for me. From the commit log : This improves the RX mbuf handling to avoid system hang due to depletion. Well... not improved for everyone ;) Sorry if the subject wasn't clear. -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
On 4/27/2011 2:45 PM, Olivier Smedts wrote: Are you testing with what is in HEAD ? ie. 7.2.3 or something else ? Your subject line implies something else. I'm using what's in HEAD since r219753, the commit which updated the em driver from version 7.1.9 to 7.2.2 and broke it at least for me. Oh, ok. Thats from back in March. There is a newer one 7.2.3 in April. I would test with that perhaps. I am using a back ported version on RELENG8 and it works quite well for me. http://lists.freebsd.org/pipermail/svn-src-head/2011-April/026202.html Hopefully MFC'd soon ? ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
Yes Mike, already have had a couple others bug me to get the MFC, I'm hoping to get it in this week :) Jack On Wed, Apr 27, 2011 at 12:04 PM, Mike Tancsa m...@sentex.net wrote: On 4/27/2011 2:45 PM, Olivier Smedts wrote: Are you testing with what is in HEAD ? ie. 7.2.3 or something else ? Your subject line implies something else. I'm using what's in HEAD since r219753, the commit which updated the em driver from version 7.1.9 to 7.2.2 and broke it at least for me. Oh, ok. Thats from back in March. There is a newer one 7.2.3 in April. I would test with that perhaps. I am using a back ported version on RELENG8 and it works quite well for me. http://lists.freebsd.org/pipermail/svn-src-head/2011-April/026202.html Hopefully MFC'd soon ? ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
problems with em(4) since update to driver 7.2.2
Hello, I've got a problem under FreeBSD 9.0-CURRENT amd64 with the em driver. It's loaded as a module. I was previously using r219710 (2011-03-17) without any problem, but with latest HEAD I can't even send a ping. Here's what appears in dmesg. I tried unloading / loading the module, did not help. At the end I loaded my old if_em.ko and it worked : em0: Intel(R) PRO/1000 Network Connection 7.2.2 port 0x2100-0x211f mem 0xf000-0xf001,0xf0025000-0xf0025fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: Could not setup receive structures em0: Could not setup receive structures em0: link state changed to UP em0: Intel(R) PRO/1000 Network Connection 7.2.2 port 0x2100-0x211f mem 0xf000-0xf001,0xf0025000-0xf0025fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: Could not setup receive structures em0: Could not setup receive structures em0: detached em0: Intel(R) PRO/1000 Network Connection 7.2.2 port 0x2100-0x211f mem 0xf000-0xf001,0xf0025000-0xf0025fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: Could not setup receive structures em0: Could not setup receive structures em0: link state changed to UP em0: detached em0: Intel(R) PRO/1000 Network Connection 7.1.9 port 0x2100-0x211f mem 0xf000-0xf001,0xf0025000-0xf0025fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: d4:85:64:b2:aa:f5 em0: link state changed to UP With 7.2.2, the Could not setup receive structures messages appeared and I had no network. The card is : em0@pci0:0:25:0:class=0x02 card=0x304b103c chip=0x10ef8086 rev=0x05 hdr=0x00 I must say I'm using clang, maybe it's related, but with a previous driver compiled with clang I don't think I had any problem. Maybe it's related to r219753 : r219753 | jfv | 2011-03-18 19:54:00 +0100 (ven 18 mar 2011) | 12 lignes This delta updates the em driver to version 7.2.2 which has been undergoing test for some weeks. This improves the RX mbuf handling to avoid system hang due to depletion. Thanks to all those who have been testing the code, and to Beezar Liu for the design changes. Next the igb driver is updated for similar RX changes, but also to add new features support for our upcoming i350 family of adapters. Can I provide anything useful to debug this ? Cheers -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email vCards X www: http://www.gid0.org - against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
On 3/31/2011 11:20 AM, Olivier Smedts wrote: Hello, I've got a problem under FreeBSD 9.0-CURRENT amd64 with the em driver. It's loaded as a module. I was previously using r219710 (2011-03-17) without any problem, but with latest HEAD I can't even send a ping. Here's what appears in dmesg. I tried unloading / loading the module, did not help. At the end I loaded my old if_em.ko and it worked : I ran into a similar problem with specific types of em NICs. For some reason, if it was compiled into the kernel it worked. As a module doing a netboot, it would fail. Does it work for you if the driver is statically compiled into the kernel by chance ? ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: problems with em(4) since update to driver 7.2.2
This problem happens for only one reason, you have insufficient mbufs to fill your rx ring. Its odd that it would differ when its static versus a loadable module though! With the 7.2.2 driver you also will use different mbuf pools depending on the MTU you are using. If you use jumbo frames it will use 4K clusters, if you go to 9K jumbos it will use 9K mbuf clusters. The number of these allocated by default is small (like 6400 small :). I would use 'netstat -m' to see what the pools look like. Now that I think about it, the reason it might fail as loaded while not as built in is you get allocation of the mbufs first when static, and something else is taking them before you can load when loadable?? I suspect all will be fine if you increase the pool size that you are using. Jack On Thu, Mar 31, 2011 at 8:52 AM, Mike Tancsa m...@sentex.net wrote: On 3/31/2011 11:20 AM, Olivier Smedts wrote: Hello, I've got a problem under FreeBSD 9.0-CURRENT amd64 with the em driver. It's loaded as a module. I was previously using r219710 (2011-03-17) without any problem, but with latest HEAD I can't even send a ping. Here's what appears in dmesg. I tried unloading / loading the module, did not help. At the end I loaded my old if_em.ko and it worked : I ran into a similar problem with specific types of em NICs. For some reason, if it was compiled into the kernel it worked. As a module doing a netboot, it would fail. Does it work for you if the driver is statically compiled into the kernel by chance ? ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
