Re: FW: ixg(4) performances
On Sun, Aug 31, 2014 at 12:07:38PM -0400, Terry Moore wrote: > > This is not 2.5G Transfers per second. PCIe talks about transactions rather > than transfers; one transaction requires either 12 bytes (for 32-bit > systems) or 16 bytes (for 64-bit systems) of overhead at the transaction > layer, plus 7 bytes at the link layer. > > The maximum number of transactions per second paradoxically transfers the > fewest number of bytes; a 4K write takes 16+4096+5+2 byte times, and so only > about 60,000 such transactions are possible per second (moving about > 248,000,000 bytes). [Real systems don't see this, quite -- Wikipedia claims, > for example 95% efficiency is typical for storage controllers.] The gain for large transfer requests is probably minimal. There can be multiple requests outstanding at any one time (the limit is negotiated, I'm guessing that 8 and 16 are typical values). A typical PCIe dma controller will generate multiple concurrent transfer requests, so even if the requests are only 128 bytes you can get a reasonable overall throughput. > A 4-byte write takes 16+4+5+2 byte times, and so roughly 9 million > transactions are possible per second, but those 9 million transactions can > only move 36 million bytes. Except that nothing will generate adequately overlapped short transfers. The real performance killer is cpu pio cycles. Every one that the driver does will hit the throughput - the cpu will be spinning for a long, long time (think ISA bus speeds). A side effect of this is that PCI-PCIe bridges (either way) are doomed to be very inefficient. > Multiple lanes scale things fairly linearly. But there has to be one byte > per lane; a x8 configuration says that physical transfers are padded so that > each the 4-byte write (which takes 27 bytes on the bus) will have to take 32 > bytes. Instead of getting 72 million transactions per second, you get 62.5 > million transactions/second, so it doesn't scale as nicely. I think that individual PCIe transfers requests always use a single lane. Multiple lanes help if you have multiple concurrent transfers. So different chunks of an ethernet frame can be transferred in parrallel over multiple lanes, with the transfer not completing until all the individual parts complete. So the ring status transfer can't be scheduled until all the other data fragment transfers have completed. I also believe that the PCIe transfers are inherently 64bit. There are byte-enables indicating which bytes of the first and last 64bit words are actually required. The real thing to remember about PCIe is that it is a comms protocol, not a bus protocol. It is high throughput, high latency. I've had 'fun' getting even moderate PCIe throughput into an fpga. David -- David Laight: da...@l8s.co.uk
Re: ixg(4) performances
On 2014/09/04 11:24, Masanobu SAITOH wrote: On 2014/09/04 0:40, Emmanuel Dreyfus wrote: On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: NetBSD 6.1 says: vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 function 0 not configured In src/sys/dev/pci/ixgbe/ we know about producct Id 0x1529 and 0x152A but not 0x1528. But this can probably be easily borrowed from FreeBSD: http://svnweb.freebsd.org/base/head/sys/dev/ixgbe/ They call it IXGBE_DEV_ID_X540T, You can try to add in ixgbe_type.h: #define IXGBE_DEV_ID_82599_X540T 0x1528 Then in ixgbe.c add a IXGBE_DEV_ID_82599_X540T line in ixgbe_vendor_info_array[] In ixgbe_82599.c you need a case IXGBE_DEV_ID_82599_X540T next to case IXGBE_DEV_ID_82599_BACKPLANE_FCOE if media is indeed backplane. Otherwise add it at the appropriate place in the switch statement. And finally you need to add a case IXGBE_DEV_ID_82599_X540T next to case IXGBE_DEV_ID_82599_BACKPLANE_FCOE in ixgbe_api.c Our ixg(4) driver doesn't support X520. At least there is no s/X520/X540/ file ixgbe/ixgbe_x540.c of FreeBSD. BusFreeBSD NetBSD 82597PCI-Xixgbdge 82598PCIeixgbeixg 82599(X520)PCIeixgbeixg X540PCIeixgbe(ixg)(not yet) -- --- SAITOH Masanobu (msai...@execsw.org msai...@netbsd.org)
Re: ixg(4) performances
On 2014/09/04 0:40, Emmanuel Dreyfus wrote: On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: NetBSD 6.1 says: vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 function 0 not configured In src/sys/dev/pci/ixgbe/ we know about producct Id 0x1529 and 0x152A but not 0x1528. But this can probably be easily borrowed from FreeBSD: http://svnweb.freebsd.org/base/head/sys/dev/ixgbe/ They call it IXGBE_DEV_ID_X540T, You can try to add in ixgbe_type.h: #define IXGBE_DEV_ID_82599_X540T 0x1528 Then in ixgbe.c add a IXGBE_DEV_ID_82599_X540T line in ixgbe_vendor_info_array[] In ixgbe_82599.c you need a case IXGBE_DEV_ID_82599_X540T next to case IXGBE_DEV_ID_82599_BACKPLANE_FCOE if media is indeed backplane. Otherwise add it at the appropriate place in the switch statement. And finally you need to add a case IXGBE_DEV_ID_82599_X540T next to case IXGBE_DEV_ID_82599_BACKPLANE_FCOE in ixgbe_api.c Our ixg(4) driver doesn't support X520. At least there is no file ixgbe/ixgbe_x540.c of FreeBSD. Bus FreeBSD NetBSD 82597 PCI-X ixgbdge 82598 PCIeixgbe ixg 82599(X520)PCIeixgbe ixg X540 PCIeixgbe (ixg)(not yet) -- --- SAITOH Masanobu (msai...@execsw.org msai...@netbsd.org)
RE: FW: ixg(4) performances
>From Emmanuel Dreyfus > You are right; > # pcictl /dev/pci5 read -d 0 -f 1 0xa8 > 00092810 > # pcictl /dev/pci5 write -d 0 -f 1 0xa8 0x00094810 > # pcictl /dev/pci5 read -d 0 -f 1 0xa8 > 4810 That's reassuring. The dump confirms that we're looking at the right registers, thank you. As I read the spec, 0x4810 in the Device Control Register means: Max_Read_Request_Size: 100b -> 4096 bytes Enable_No_Snoop: 1 Max_Payload_Size: 000b --> 128 bytes Enable_Relaxed_Ordering: 1 All other options turned off. I think you should try: Max_Read_Request_Size: 100b -> 4096 bytes Enable_No_Snoop: 1 Max_Payload_Size: 100b --> 4096 bytes Enable_Relaxed_Ordering: 1 This would give 0x4890 as the value, not 0x4810. It's odd that the BIOS set the max_payload_size to 000b. It's possible that this indicates that the root complex has some limitations. Or it could be a buggy or excessively conservative BIOS. ("It's safer to program Add-In boards conservatively -- fewer support calls due to dead systems." Or something like that.) So you may have to experiment. This would explain that you saw 2.5 GB/sec before, and 2.7 GB/sec after -- you increased the max *read* size, but not the max *write* size. Increasing from 2048 to 4096 would improve read throughput but not enormously. Depends, of course, on your benchmark. --Terry
RE: FW: ixg(4) performances
> -Original Message- > From: Hisashi T Fujinaka [mailto:ht...@twofifty.com] > Sent: Sunday, August 31, 2014 12:39 > To: Terry Moore > Cc: tech-kern@netbsd.org > Subject: RE: FW: ixg(4) performances > > I may be wrong in the transactions/transfers. However, I think you're > reading the page incorrectly. The signalling rate is the physical speed of > the link. On top of that is the 8/10 encoding (the Ethernet controller > we're talking about is only Gen 2), the framing, etc, and the spec > discusses the data rate in GT/s. Gb/s means nothing. Hi, I see what the dispute is. The PCIe 3.0 spec *nowhere* uses "transfers" in the sense of UNIT INTERVALs (it uses UNIT INTERVAL). The word "transfer" is not used in that way in the spec. Transfer is used, mostly in the sense of a larger message. It's not in the glossary, etc. However, you are absolutely right, many places in the industry (Intel, Wikipedia, etc.) refer to 2.5GT/s to mean 2.5G Unit Intervals/Second; and it's common enough that it's the de-facto standard terminology. I only deal with the spec, and don't pay that much attention to the ancillary material. We still disagree about something: GT/s does mean *something* in terms of the raw throughput of the link. It tells the absolute upper limit of the channel (ignoring protocol overhead). If the upper limit is too low, you can stop worrying about fine points. If you're only trying to push 10% (for example) and you're not getting it, you look for protocol problems. At 50%, you say "hmm, we could be hitting the channel capacity". I think we were violently agreeing, because the technical content of what I was writing was (modulo possible typos) identical to 2.5GT/s. 2.5GT/s is (after 8/10 encoding) a max raw data rate of 250e6 bytes/sec, and when you go through things and account for overhead, it's very possible that 8 lanes (max 2e9 bytes/sec) of gen1 won't be fast enough for 10G Eth. Best regards, --Terry
Re: ixg(4) performances
On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: > NetBSD 6.1 says: > vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 > function 0 not configured In src/sys/dev/pci/ixgbe/ we know about producct Id 0x1529 and 0x152A but not 0x1528. But this can probably be easily borrowed from FreeBSD: http://svnweb.freebsd.org/base/head/sys/dev/ixgbe/ They call it IXGBE_DEV_ID_X540T, You can try to add in ixgbe_type.h: #define IXGBE_DEV_ID_82599_X540T 0x1528 Then in ixgbe.c add a IXGBE_DEV_ID_82599_X540T line in ixgbe_vendor_info_array[] In ixgbe_82599.c you need a case IXGBE_DEV_ID_82599_X540T next to case IXGBE_DEV_ID_82599_BACKPLANE_FCOE if media is indeed backplane. Otherwise add it at the appropriate place in the switch statement. And finally you need to add a case IXGBE_DEV_ID_82599_X540T next to case IXGBE_DEV_ID_82599_BACKPLANE_FCOE in ixgbe_api.c -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: > NetBSD 6.1 says: > > vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 > function 0 not configured > > Complete messages: http://netbsd.itsx.net/hw/x9drw.dmesg > > NetBSD current from today also does not configure them. Btw, FreeBSD 9.3-RELEASE says: ix0: port 0x8020-0x803f mem 0xde20-0xde3f,0xde404000-0xde407fff irq 26 at device 0.0 on pci1 ix0: Using MSIX interrupts with 9 vectors ix0: Ethernet address: 00:25:90:f9:49:20 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 -- B*E*R*T
Re: ixg(4) performances
On Sat, Aug 30, 2014 at 10:24:52AM +0100, Justin Cormack wrote: > On Sat, Aug 30, 2014 at 8:22 AM, Thor Lancelot Simon wrote: > > On Fri, Aug 29, 2014 at 12:22:31PM -0400, Terry Moore wrote: > >> > >> Is the ixg in an expansion slot or integrated onto the main board? > > > > If you know where to get a mainboard with an integrated ixg, I wouldn't > > mind hearing about it. > > They are starting to appear, eg > http://www.supermicro.co.uk/products/motherboard/Xeon/C600/X9SRH-7TF.cfm We have some SuperMicro X9DRW with 10 GbE Intel NICs on board. NetBSD current does not configure them. NetBSD 6.1 says: vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 function 0 not configured Complete messages: http://netbsd.itsx.net/hw/x9drw.dmesg NetBSD current from today also does not configure them. (I boot from an USB-stick with current kernel and 6.1 userland. It wants to mount sda0 but there is only dk0, dk1. So I end up with read/only disk. I have to sort that out before I can save kernels messages.) Grtnx, -- B*E*R*T
Re: FW: ixg(4) performances
Hi, Emmanuel. On 2014/09/01 11:10, Emmanuel Dreyfus wrote: > Terry Moore wrote: > >> Since you did a dword read, the extra 0x9 is the device status register. >> This makes me suspicious as the device status register is claiming that you >> have "unsupported request detected)" [bit 3] and "correctable error >> detected" [bit 0]. Further, this register is RW1C for all these bits -- so >> when you write 94810, it should have cleared the 9 (so a subsequent read >> should have returned 4810). >> >> Please check. > > You are right; > # pcictl /dev/pci5 read -d 0 -f 1 0xa8 > 00092810 > # pcictl /dev/pci5 write -d 0 -f 1 0xa8 0x00094810 > # pcictl /dev/pci5 read -d 0 -f 1 0xa8 > 4810 > >> Might be good to post a "pcictl dump" of your device, just to expose all the >> details. > > It explicitely says 2.5 Gb/s x 8 lanes > > # pcictl /dev/pci5 dump -d0 -f 1 > PCI configuration registers: >Common header: > 0x00: 0x10fb8086 0x00100107 0x0201 0x00800010 > > Vendor Name: Intel (0x8086) > Device Name: 82599 (SFI/SFP+) 10 GbE Controller (0x10fb) > Command register: 0x0107 >I/O space accesses: on >Memory space accesses: on >Bus mastering: on >Special cycles: off >MWI transactions: off >Palette snooping: off >Parity error checking: off >Address/data stepping: off >System error (SERR): on >Fast back-to-back transactions: off >Interrupt disable: off > Status register: 0x0010 >Interrupt status: inactive >Capability List support: on >66 MHz capable: off >User Definable Features (UDF) support: off >Fast back-to-back capable: off >Data parity error detected: off >DEVSEL timing: fast (0x0) >Slave signaled Target Abort: off >Master received Target Abort: off >Master received Master Abort: off >Asserted System Error (SERR): off >Parity error detected: off > Class Name: network (0x02) > Subclass Name: ethernet (0x00) > Interface: 0x00 > Revision ID: 0x01 > BIST: 0x00 > Header Type: 0x00+multifunction (0x80) > Latency Timer: 0x00 > Cache Line Size: 0x10 > >Type 0 ("normal" device) header: > 0x10: 0xdfe8000c 0x 0xbc01 0x > 0x20: 0xdfe7c00c 0x 0x 0x00038086 > 0x30: 0x 0x0040 0x 0x0209 > > Base address register at 0x10 >type: 64-bit prefetchable memory >base: 0xdfe8, not sized > Base address register at 0x18 >type: i/o >base: 0xbc00, not sized > Base address register at 0x1c >not implemented(?) > Base address register at 0x20 >type: 64-bit prefetchable memory >base: 0xdfe7c000, not sized > Cardbus CIS Pointer: 0x > Subsystem vendor ID: 0x8086 > Subsystem ID: 0x0003 > Expansion ROM Base Address: 0x > Capability list pointer: 0x40 > Reserved @ 0x38: 0x > Maximum Latency: 0x00 > Minimum Grant: 0x00 > Interrupt pin: 0x02 (pin B) > Interrupt line: 0x09 > >Capability register at 0x40 > type: 0x01 (Power Management, rev. 1.0) >Capability register at 0x50 > type: 0x05 (MSI) >Capability register at 0x70 > type: 0x11 (MSI-X) >Capability register at 0xa0 > type: 0x10 (PCI Express) > >PCI Message Signaled Interrupt > Message Control register: 0x0180 >MSI Enabled: no >Multiple Message Capable: no (1 vector) >Multiple Message Enabled: off (1 vector) >64 Bit Address Capable: yes >Per-Vector Masking Capable: yes > Message Address (lower) register: 0x > Message Address (upper) register: 0x > Message Data register: 0x > Vector Mask register: 0x > Vector Pending register: 0x > >PCI Power Management Capabilities Register > Capabilities register: 0x4823 >Version: 1.2 >PME# clock: off >Device specific initialization: on >3.3V auxiliary current: self-powered >D1 power management state support: off >D2 power management state support: off >PME# support: 0x09 > Control/status register: 0x2000 >Power state: D0 >PCI Express reserved: off >No soft reset: off >PME# assertion disabled >PME# status: off > >PCI Express Capabilities Register > Capability version: 2 > Device type: PCI Express Endpoint device > Interrupt Message Number: 0 > Link Capabilities Register: 0x00027482 >Maximum Link Speed: unknown 2 value >Maximum Link Width: x8 lanes >Port Number: 0 > Link Status Register: 0x1081 >Negotiated Link Speed: 2.5Gb/s * Which Version of NetBSD are you using? I committed some changes fixing Gb/s
Re: FW: ixg(4) performances
Terry Moore wrote: > Since you did a dword read, the extra 0x9 is the device status register. > This makes me suspicious as the device status register is claiming that you > have "unsupported request detected)" [bit 3] and "correctable error > detected" [bit 0]. Further, this register is RW1C for all these bits -- so > when you write 94810, it should have cleared the 9 (so a subsequent read > should have returned 4810). > > Please check. You are right; # pcictl /dev/pci5 read -d 0 -f 1 0xa8 00092810 # pcictl /dev/pci5 write -d 0 -f 1 0xa8 0x00094810 # pcictl /dev/pci5 read -d 0 -f 1 0xa8 4810 > Might be good to post a "pcictl dump" of your device, just to expose all the > details. It explicitely says 2.5 Gb/s x 8 lanes # pcictl /dev/pci5 dump -d0 -f 1 PCI configuration registers: Common header: 0x00: 0x10fb8086 0x00100107 0x0201 0x00800010 Vendor Name: Intel (0x8086) Device Name: 82599 (SFI/SFP+) 10 GbE Controller (0x10fb) Command register: 0x0107 I/O space accesses: on Memory space accesses: on Bus mastering: on Special cycles: off MWI transactions: off Palette snooping: off Parity error checking: off Address/data stepping: off System error (SERR): on Fast back-to-back transactions: off Interrupt disable: off Status register: 0x0010 Interrupt status: inactive Capability List support: on 66 MHz capable: off User Definable Features (UDF) support: off Fast back-to-back capable: off Data parity error detected: off DEVSEL timing: fast (0x0) Slave signaled Target Abort: off Master received Target Abort: off Master received Master Abort: off Asserted System Error (SERR): off Parity error detected: off Class Name: network (0x02) Subclass Name: ethernet (0x00) Interface: 0x00 Revision ID: 0x01 BIST: 0x00 Header Type: 0x00+multifunction (0x80) Latency Timer: 0x00 Cache Line Size: 0x10 Type 0 ("normal" device) header: 0x10: 0xdfe8000c 0x 0xbc01 0x 0x20: 0xdfe7c00c 0x 0x 0x00038086 0x30: 0x 0x0040 0x 0x0209 Base address register at 0x10 type: 64-bit prefetchable memory base: 0xdfe8, not sized Base address register at 0x18 type: i/o base: 0xbc00, not sized Base address register at 0x1c not implemented(?) Base address register at 0x20 type: 64-bit prefetchable memory base: 0xdfe7c000, not sized Cardbus CIS Pointer: 0x Subsystem vendor ID: 0x8086 Subsystem ID: 0x0003 Expansion ROM Base Address: 0x Capability list pointer: 0x40 Reserved @ 0x38: 0x Maximum Latency: 0x00 Minimum Grant: 0x00 Interrupt pin: 0x02 (pin B) Interrupt line: 0x09 Capability register at 0x40 type: 0x01 (Power Management, rev. 1.0) Capability register at 0x50 type: 0x05 (MSI) Capability register at 0x70 type: 0x11 (MSI-X) Capability register at 0xa0 type: 0x10 (PCI Express) PCI Message Signaled Interrupt Message Control register: 0x0180 MSI Enabled: no Multiple Message Capable: no (1 vector) Multiple Message Enabled: off (1 vector) 64 Bit Address Capable: yes Per-Vector Masking Capable: yes Message Address (lower) register: 0x Message Address (upper) register: 0x Message Data register: 0x Vector Mask register: 0x Vector Pending register: 0x PCI Power Management Capabilities Register Capabilities register: 0x4823 Version: 1.2 PME# clock: off Device specific initialization: on 3.3V auxiliary current: self-powered D1 power management state support: off D2 power management state support: off PME# support: 0x09 Control/status register: 0x2000 Power state: D0 PCI Express reserved: off No soft reset: off PME# assertion disabled PME# status: off PCI Express Capabilities Register Capability version: 2 Device type: PCI Express Endpoint device Interrupt Message Number: 0 Link Capabilities Register: 0x00027482 Maximum Link Speed: unknown 2 value Maximum Link Width: x8 lanes Port Number: 0 Link Status Register: 0x1081 Negotiated Link Speed: 2.5Gb/s Negotiated Link Width: x8 lanes Device-dependent header: 0x40: 0x48235001 0x2b002000 0x 0x 0x50: 0x01807005 0x 0x 0x 0x60: 0x 0x 0x 0x 0x70: 0x003fa011 0x0004 0x2004 0x 0x80: 0x 0x 0x 0x 0x90: 0x 0x 0x 0x 0xa0: 0x00020010 0x10008cc2 0x4810 0x00027482 0xb0: 0x1081 0x 0x 0x 0xc0: 0x 0x001f 0x000
RE: FW: ixg(4) performances
Oh, and to answer the actual first, relevant question, I can try finding out if we (day job, 82599) can do line rate at 2.5GT/s. I think we can get a lot closer than you're getting but we don't test with NetBSD. -- Hisashi T Fujinaka - ht...@twofifty.com BSEE + BSChem + BAEnglish + MSCS + $2.50 = coffee
RE: FW: ixg(4) performances
I may be wrong in the transactions/transfers. However, I think you're reading the page incorrectly. The signalling rate is the physical speed of the link. On top of that is the 8/10 encoding (the Ethernet controller we're talking about is only Gen 2), the framing, etc, and the spec discusses the data rate in GT/s. Gb/s means nothing. It's like talking about the frequency of the Ethernet link, which we never do. We talk about how much data can be transferred. I'm also not sure if you've looked at an actual trace before, but a PCIe link is incredibly chatty, and every transfer only has a payload of 64/128/256b (especially regarding the actual controller again). So, those two coupled together (GT/s & chatty link with small packets) means talking about things in Gb/s is not something used by people who talk about PCIe every day (my day job). The signalling rate is not used when talking about the max data transfer rate. On Sun, 31 Aug 2014, Terry Moore wrote: -Original Message- From: Hisashi T Fujinaka [mailto:ht...@twofifty.com] Sent: Saturday, August 30, 2014 21:29 To: Terry Moore Cc: tech-kern@netbsd.org Subject: Re: FW: ixg(4) performances Doesn't anyone read my posts or, more important, the PCIe spec? 2.5 Giga TRANSFERS per second. I'm not sure I understand what you're saying. From the PCIe space, page 40: "Signaling rate - Once initialized, each Link must only operate at one of the supported signaling levels. For the first generation of PCI Express technology, there is only one signaling rate defined, which provides an effective 2.5 Gigabits/second/Lane/direction of raw bandwidth. The second generation provides an effective 5.0 Gigabits/second/Lane/direction of raw bandwidth. The third generation provides an effective 8.0 Gigabits/second/Lane/direction of 10 raw bandwidth. The data rate is expected to increase with technology advances in the future." This is not 2.5G Transfers per second. PCIe talks about transactions rather than transfers; one transaction requires either 12 bytes (for 32-bit systems) or 16 bytes (for 64-bit systems) of overhead at the transaction layer, plus 7 bytes at the link layer. The maximum number of transactions per second paradoxically transfers the fewest number of bytes; a 4K write takes 16+4096+5+2 byte times, and so only about 60,000 such transactions are possible per second (moving about 248,000,000 bytes). [Real systems don't see this, quite -- Wikipedia claims, for example 95% efficiency is typical for storage controllers.] A 4-byte write takes 16+4+5+2 byte times, and so roughly 9 million transactions are possible per second, but those 9 million transactions can only move 36 million bytes. Multiple lanes scale things fairly linearly. But there has to be one byte per lane; a x8 configuration says that physical transfers are padded so that each the 4-byte write (which takes 27 bytes on the bus) will have to take 32 bytes. Instead of getting 72 million transactions per second, you get 62.5 million transactions/second, so it doesn't scale as nicely. Reads are harder to analyze, because they depend on the speed and design of both ends of the link. The reader sends a read request packet, and the read-responder (some time later) sends back the response. As far as I can see, even at gen3 with lots of lanes, PCIe doesn't scale to 2.5 G transfers per second. Best regards, --Terry -- Hisashi T Fujinaka - ht...@twofifty.com BSEE + BSChem + BAEnglish + MSCS + $2.50 = coffee
RE: FW: ixg(4) performances
> -Original Message- > From: Hisashi T Fujinaka [mailto:ht...@twofifty.com] > Sent: Saturday, August 30, 2014 21:29 > To: Terry Moore > Cc: tech-kern@netbsd.org > Subject: Re: FW: ixg(4) performances > > Doesn't anyone read my posts or, more important, the PCIe spec? > > 2.5 Giga TRANSFERS per second. I'm not sure I understand what you're saying. >From the PCIe space, page 40: "Signaling rate - Once initialized, each Link must only operate at one of the supported signaling levels. For the first generation of PCI Express technology, there is only one signaling rate defined, which provides an effective 2.5 Gigabits/second/Lane/direction of raw bandwidth. The second generation provides an effective 5.0 Gigabits/second/Lane/direction of raw bandwidth. The third generation provides an effective 8.0 Gigabits/second/Lane/direction of 10 raw bandwidth. The data rate is expected to increase with technology advances in the future." This is not 2.5G Transfers per second. PCIe talks about transactions rather than transfers; one transaction requires either 12 bytes (for 32-bit systems) or 16 bytes (for 64-bit systems) of overhead at the transaction layer, plus 7 bytes at the link layer. The maximum number of transactions per second paradoxically transfers the fewest number of bytes; a 4K write takes 16+4096+5+2 byte times, and so only about 60,000 such transactions are possible per second (moving about 248,000,000 bytes). [Real systems don't see this, quite -- Wikipedia claims, for example 95% efficiency is typical for storage controllers.] A 4-byte write takes 16+4+5+2 byte times, and so roughly 9 million transactions are possible per second, but those 9 million transactions can only move 36 million bytes. Multiple lanes scale things fairly linearly. But there has to be one byte per lane; a x8 configuration says that physical transfers are padded so that each the 4-byte write (which takes 27 bytes on the bus) will have to take 32 bytes. Instead of getting 72 million transactions per second, you get 62.5 million transactions/second, so it doesn't scale as nicely. Reads are harder to analyze, because they depend on the speed and design of both ends of the link. The reader sends a read request packet, and the read-responder (some time later) sends back the response. As far as I can see, even at gen3 with lots of lanes, PCIe doesn't scale to 2.5 G transfers per second. Best regards, --Terry
Re: FW: ixg(4) performances
Doesn't anyone read my posts or, more important, the PCIe spec? 2.5 Giga TRANSFERS per second. On Sat, 30 Aug 2014, Terry Moore wrote: Forgot to cc the list. -Original Message- From: Terry Moore [mailto:t...@mcci.com] Sent: Friday, August 29, 2014 15:13 To: 'Emmanuel Dreyfus' Subject: RE: ixg(4) performances But it's running at gen1. I strongly suspect that the benchmark case was gen2 (since the ixg is capable of it). gen1 vs gen2 is 2.5 Gb.s vs 5 Gb/s? Yes. Actually, 2.5Gbps is symbol rate -- it's 8/10 encoded, so one lane is really 2Gbps. So 8 lanes is 16Gbps which *should* be enough, but... there's overhead and a variety of sources of wastage. I just saw today a slide that says that 8 lanes gen 1 is just barely enough for 10Gb Eth (http://www.eetimes.com/document.asp?doc_id=1323695&page_number=4). It's possible that the benchmark system was using 8 lanes of Gen2. Another possibility: earlier you wrote: No reference to MMRBC in this document, but I understand "Max Read Request Size" is the same thing. Page 765 tells us about register A8, bits 12-14 that should be set to 100. pcictl /dev/pci5 read -d 0 -f 1 0x18 tells me the value 0x00092810 I tried this command: pcictl /dev/pci5 write -d 0 -f 1 0x18 0x00094810 In the PCIe spec, this is controlled via the Device Control register which is a 16-bit value. You want to set *two* fields, to different values. 0x18 looks like the wrong offset. The PCIe spec says offset 0x08, but that's relative to the base of the capability structure, the offset of which is in the low byte of the dword at 0x34. I am running NetBSD 5; my pcictl doesn't support write as one of its options, but I'd expect that to be relative to the base of the function config space, and *that's not the device capabilities register*. It's a read/write register, which is one of the Base Address Registers. In any case, in the Device Control Register, bits 7:5 are the max payload size *for writes* by the igx to the system. These must be set to 101b for 4K max payload size. Similarly, bits 14:12 are the max read request size. These must also be set to 101b for 4K max read request. Since you did a dword read, the extra 0x9 is the device status register. This makes me suspicious as the device status register is claiming that you have "unsupported request detected)" [bit 3] and "correctable error detected" [bit 0]. Further, this register is RW1C for all these bits -- so when you write 94810, it should have cleared the 9 (so a subsequent read should have returned 4810). Please check. Might be good to post a "pcictl dump" of your device, just to expose all the details. Is the ixg in an expansion slot or integrated onto the main board? In a slot. Check the manual on the main board and find out whether other slots have 8 lanes of Gen2. If so, move the board. Best regards, --Terry -- Hisashi T Fujinaka - ht...@twofifty.com BSEE + BSChem + BAEnglish + MSCS + $2.50 = coffee
FW: ixg(4) performances
Forgot to cc the list. -Original Message- From: Terry Moore [mailto:t...@mcci.com] Sent: Friday, August 29, 2014 15:13 To: 'Emmanuel Dreyfus' Subject: RE: ixg(4) performances > > But it's running at gen1. I strongly suspect that the benchmark case > > was > > gen2 (since the ixg is capable of it). > > gen1 vs gen2 is 2.5 Gb.s vs 5 Gb/s? Yes. Actually, 2.5Gbps is symbol rate -- it's 8/10 encoded, so one lane is really 2Gbps. So 8 lanes is 16Gbps which *should* be enough, but... there's overhead and a variety of sources of wastage. I just saw today a slide that says that 8 lanes gen 1 is just barely enough for 10Gb Eth (http://www.eetimes.com/document.asp?doc_id=1323695&page_number=4). It's possible that the benchmark system was using 8 lanes of Gen2. Another possibility: earlier you wrote: > No reference to MMRBC in this document, but I understand "Max Read Request Size" > is the same thing. Page 765 tells us about register A8, bits 12-14 > that should be set to 100. > pcictl /dev/pci5 read -d 0 -f 1 0x18 tells me the value 0x00092810 > > I tried this command: > pcictl /dev/pci5 write -d 0 -f 1 0x18 0x00094810 In the PCIe spec, this is controlled via the Device Control register which is a 16-bit value. You want to set *two* fields, to different values. 0x18 looks like the wrong offset. The PCIe spec says offset 0x08, but that's relative to the base of the capability structure, the offset of which is in the low byte of the dword at 0x34. I am running NetBSD 5; my pcictl doesn't support write as one of its options, but I'd expect that to be relative to the base of the function config space, and *that's not the device capabilities register*. It's a read/write register, which is one of the Base Address Registers. In any case, in the Device Control Register, bits 7:5 are the max payload size *for writes* by the igx to the system. These must be set to 101b for 4K max payload size. Similarly, bits 14:12 are the max read request size. These must also be set to 101b for 4K max read request. Since you did a dword read, the extra 0x9 is the device status register. This makes me suspicious as the device status register is claiming that you have "unsupported request detected)" [bit 3] and "correctable error detected" [bit 0]. Further, this register is RW1C for all these bits -- so when you write 94810, it should have cleared the 9 (so a subsequent read should have returned 4810). Please check. Might be good to post a "pcictl dump" of your device, just to expose all the details. > > Is the ixg in an expansion slot or integrated onto the main board? > > In a slot. Check the manual on the main board and find out whether other slots have 8 lanes of Gen2. If so, move the board. Best regards, --Terry
Re: ixg(4) performances
Matthias Drochner wrote: > > Link Capabilities Ragister (0xAC): 0x00027482 > > bits 3:0 Supprted Link speed: 0010 = 5 GbE and 2.5 GbE speed > > supported bits 9:4 Max link width: 001000 = x4 > > Wrong -- this means x8. > > > bits 14:12 L0s exit lattency: 101 = 1 µs - 2 µs > > bits 17:15 L1 Exit lattency: 011 = 4 µs - 8 µs > > > > Link Status Register (0xB2): 0x1081 > > bits 3:0 Current Link speed: 0001 = 2.5 GbE PCIe link > > bits 9:4 Negociated link width: 001000 = x8 > > So it makes sense. Right, hence the ethernet board can do 5 GbE x 8, but something in front of it can only do 2.5 GbE x 8. But 2.5 GbE x 8 means 20 Gb/s, which is much larger than the 2.7 Gb/s I get. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org
Re: ixg(4) performances
On Fri, 29 Aug 2014 15:51:14 + Emmanuel Dreyfus wrote: > I found this, but the result does not make sense: negociated > max ... > > Link Capabilities Ragister (0xAC): 0x00027482 > bits 3:0 Supprted Link speed: 0010 = 5 GbE and 2.5 GbE speed > supported bits 9:4 Max link width: 001000 = x4 Wrong -- this means x8. > bits 14:12 L0s exit lattency: 101 = 1 µs - 2 µs > bits 17:15 L1 Exit lattency: 011 = 4 µs - 8 µs > > Link Status Register (0xB2): 0x1081 > bits 3:0 Current Link speed: 0001 = 2.5 GbE PCIe link > bits 9:4 Negociated link width: 001000 = x8 So it makes sense. best regards Matthias Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt
Re: ixg(4) performances
On Sat, Aug 30, 2014 at 8:22 AM, Thor Lancelot Simon wrote: > On Fri, Aug 29, 2014 at 12:22:31PM -0400, Terry Moore wrote: >> >> Is the ixg in an expansion slot or integrated onto the main board? > > If you know where to get a mainboard with an integrated ixg, I wouldn't > mind hearing about it. They are starting to appear, eg http://www.supermicro.co.uk/products/motherboard/Xeon/C600/X9SRH-7TF.cfm Justin
Re: ixg(4) performances
On Fri, Aug 29, 2014 at 12:22:31PM -0400, Terry Moore wrote: > > Is the ixg in an expansion slot or integrated onto the main board? If you know where to get a mainboard with an integrated ixg, I wouldn't mind hearing about it. Thor
Re: ixg(4) performances
On Fri, 29 Aug 2014, Emmanuel Dreyfus wrote: Terry Moore wrote: But it's running at gen1. I strongly suspect that the benchmark case was gen2 (since the ixg is capable of it). gen1 vs gen2 is 2.5 Gb.s bs 5 Gb/s? Gen 1 is capable of only 2.5GT/s (gigatransfers per second). Gen 2 is capable of up to 5, but isn't guaranteed to be 5. Depending on how chatty the device is on the PCIe bus, I think 2.5GT/s is enough for something much closer to line rate than you're getting. -- Hisashi T Fujinaka - ht...@twofifty.com BSEE + BSChem + BAEnglish + MSCS + $2.50 = coffee
Re: ixg(4) performances
Terry Moore wrote: > But it's running at gen1. I strongly suspect that the benchmark case was > gen2 (since the ixg is capable of it). gen1 vs gen2 is 2.5 Gb.s bs 5 Gb/s? > Is the ixg in an expansion slot or integrated onto the main board? In a slot. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org
RE: ixg(4) performances
> On Friday, August 29, 2014 11:51, Emmanuel Dreyfus wrote: > > On Fri, Aug 29, 2014 at 08:48:51AM -0400, Terry Moore wrote: > > Still, you should check whether you have the right number of the right > > generation of PCIe lanes connected to the ixg. > > I found this, but the result does not make sense: negociated > max ... > > Link Capabilities Ragister (0xAC): 0x00027482 > bits 3:0 Supprted Link speed: 0010 = 5 GbE and 2.5 GbE speed supported > bits 9:4 Max link width: 001000 = x4 > bits 14:12 L0s exit lattency: 101 = 1 µs - 2 µs bits 17:15 L1 Exit > lattency: 011 = 4 µs - 8 µs > > Link Status Register (0xB2): 0x1081 > bits 3:0 Current Link speed: 0001 = 2.5 GbE PCIe link > bits 9:4 Negociated link width: 001000 = x8 I think there's a typo in the docs. In the PCIe spec, it says (for Link Capabilities Register, table 7-15): 001000b is x8. But it's running at gen1. I strongly suspect that the benchmark case was gen2 (since the ixg is capable of it). Is the ixg in an expansion slot or integrated onto the main board? --Terry
Re: ixg(4) performances
On Fri, Aug 29, 2014 at 08:48:51AM -0400, Terry Moore wrote: > Still, you should check whether you have the right number of the right > generation of PCIe lanes connected to the ixg. I found this, but the result does not make sense: negociated > max ... Link Capabilities Ragister (0xAC): 0x00027482 bits 3:0 Supprted Link speed: 0010 = 5 GbE and 2.5 GbE speed supported bits 9:4 Max link width: 001000 = x4 bits 14:12 L0s exit lattency: 101 = 1 µs - 2 µs bits 17:15 L1 Exit lattency: 011 = 4 µs - 8 µs Link Status Register (0xB2): 0x1081 bits 3:0 Current Link speed: 0001 = 2.5 GbE PCIe link bits 9:4 Negociated link width: 001000 = x8 -- Emmanuel Dreyfus m...@netbsd.org
RE: ixg(4) performances
> -Original Message- > From: tech-kern-ow...@netbsd.org [mailto:tech-kern-ow...@netbsd.org] On > Behalf Of Emmanuel Dreyfus > Sent: Thursday, August 28, 2014 23:55 > To: Terry Moore; 'Christos Zoulas' > Cc: tech-kern@netbsd.org > Subject: Re: ixg(4) performances > > Terry Moore wrote: > > > There are several possibilities, all revolving about differences > > between the blog poster's base system and yorus. > > Do I have a way to investigate for appropriate PCI setup? Here is what > dmesg says about it: > > pci0 at mainbus0 bus 0: configuration mode 1 > pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok > ppb4 at pci0 dev 14 function 0: vendor 0x10de product 0x005d (rev. 0xa3) > ppb4: PCI Express 1.0 > pci5 at ppb4 bus 5 > pci5: i/o space, memory space enabled, rd/line, wr/inv ok > ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network > Driver, Version - 2.3.10 I don't do PCIe on NetBSD -- these days we use it exclusively as VM guests --, so I don't know what tools are available. Normally when doing this kind of thing I poke around with a debugger or equivalent of pcictl. The dmesg output tells us that your ixg is directly connected to an Nvdia root complex. So there are no bridges, but this might be a relevant difference to the benchmark system. It's more common to be connected to an Intel southbridge chip of some kind. Next step would be to check the documentation on, and the configuration of, the root complex -- it must also be configured for 4K read ahead (because the read will launch from the ixg, will be buffered in the root complex, forwarded to the memory controller, and then the answers will come back. (PCIe is very fast at the bit transfer level, but pretty slow in terms of read transfers per second. Read transfer latency is on the order of 2.5 microseconds / operation. This is why 4K transfers are so important in this application.) Anyway, there are multiple vendors involved (Intel, Nvidia, your BIOS maker, because the BIOS is responsible for setting things like the read size to the maximum across the bus -- I'm speaking loosely, but basically config software has to set things up because the individual devices don't have enough knowledge). So generally that may explain things. Still, you should check whether you have the right number of the right generation of PCIe lanes connected to the ixg. If you look at the manual, normally there's an obscure register that tells you how many lanes are connected, and what generation. On the motherboards we use, each slot is different, and it's not always obvious how the slots differ. Rather than depending on documentation and the good intentions of the motherboard developers, I always feel better looking at what the problem chip in question thinks about number of lanes and speeds. Hope this helps, --Terry
Re: ixg(4) performances
Terry Moore wrote: > There are several possibilities, all revolving about differences between the > blog poster's base system and yorus. Do I have a way to investigate for appropriate PCI setup? Here is what dmesg says about it: pci0 at mainbus0 bus 0: configuration mode 1 pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok ppb4 at pci0 dev 14 function 0: vendor 0x10de product 0x005d (rev. 0xa3) ppb4: PCI Express 1.0 pci5 at ppb4 bus 5 pci5: i/o space, memory space enabled, rd/line, wr/inv ok ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10 -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org
Re: ixg(4) performances
On Thu, Aug 28, 2014 at 08:37:06AM -0700, Hisashi T Fujinaka wrote: > Isn't your PCIe slot constrained? I thought I remembered that you're > only getting 2.5GT/s and I forget what test you're running. I use netperf, and I now get 2.7 Gb/s. -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
On Thu, 28 Aug 2014, Emmanuel Dreyfus wrote: On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: I also found this page that tackles the same problem on Linux: http://dak1n1.com/blog/7-performance-tuning-intel-10gbe It seems that page describe a slightly different model. Intel 82599 datasheet is available here: http://www.intel.fr/content/www/fr/fr/ethernet-controllers/82599-10-gbe-controller-datasheet.html No reference to MMRBC in this document, but I understand "Max Read Request Size" is the same thing. Page 765 tells us about register A8, bits 12-14 that should be set to 100. pcictl /dev/pci5 read -d 0 -f 1 0x18 tells me the value 0x00092810 I tried this command: pcictl /dev/pci5 write -d 0 -f 1 0x18 0x00094810 Further pcictl read suggests it works as the new value is returned. However it gives no performance improvement. This means that I misunderstood what this register is about, or how to change it (byte order?). Or the performance are constrained by something unrelated. In the blog post cited above, the poster acheived more than 5 Gb/s before touching MMRBC, while I am stuck at 2,7 GB/s. Any new idea welcome. Isn't your PCIe slot constrained? I thought I remembered that you're only getting 2.5GT/s and I forget what test you're running. -- Hisashi T Fujinaka - ht...@twofifty.com BSEE + BSChem + BAEnglish + MSCS + $2.50 = coffee
Re: ixg(4) performances
In article <20140828072832.gi8...@homeworld.netbsd.org>, Emmanuel Dreyfus wrote: >On Tue, Aug 26, 2014 at 04:40:25PM +, Taylor R Campbell wrote: >> New version with some changes suggested by wiz@. > >Anyone has objection to this change being committed and pulled up to >netbsd-7? > Not me. christos
RE: ixg(4) performances
> Or the performance are constrained by something unrelated. In the blog > post cited above, the poster acheived more than 5 Gb/s before touching > MMRBC, while I am stuck at 2,7 GB/s. Any new idea welcome. The blog post refers to PCI-X, I'm more familiar with PCIe, but the concepts are similar. There are several possibilities, all revolving about differences between the blog poster's base system and yorus. 1) the test case is using a platform that has better PCI performance (in the PCIe world this could be: Gen3 versus Gen2 support in the slot being used; more lanes in the slot being used) 2) the test case has a root complex with a PCI controller with better performance than the one in your system; 3) the test case system has a different PCI configuration, in particular different bridging. For example, a PCI bridge or switch on your platform can change basic capabilities compared to the reference. 4) related to 3: one of the bridges on your system (between ixg and root complex) is not configured for 4K reads, and so the setting on the ixg board won't help [whereas this wasn't the case on the blog system]. 5) related to 4: one of the bridges in your system (between ixg and root complex) is not capable of 4K reads... (see 4). And of course you have to consider: 6) the writer has something else different than you have, for example silicon rev, BIOS, PCI-X where you have PCIe, etc. 7) the problem is completely unrelated to PCIe. You're in a tough situation, experimentally, because you can't take a working (5 Gbps) system and directly compare to the non-working (2.7 Gbps) situation. --Terry
Re: ixg(4) performances
What is your test setup? Do you have 2 identical boxes? Does it perform better e.g. on Linux or FreeBSD? If so, you could check how the config registers get set by that particular OS. 2014-08-28 9:26 GMT+02:00 Emmanuel Dreyfus : > On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: >> >I also found this page that tackles the same problem on Linux: >> >http://dak1n1.com/blog/7-performance-tuning-intel-10gbe > > It seems that page describe a slightly different model. > Intel 82599 datasheet is available here: > http://www.intel.fr/content/www/fr/fr/ethernet-controllers/82599-10-gbe-controller-datasheet.html > > No reference to MMRBC in this document, but I understand "Max Read Request > Size" is the same thing. Page 765 tells us about register A8, bits 12-14 > that should be set to 100. > pcictl /dev/pci5 read -d 0 -f 1 0x18 tells me the value 0x00092810 > > I tried this command: > pcictl /dev/pci5 write -d 0 -f 1 0x18 0x00094810 > > Further pcictl read suggests it works as the new value is returned. > However it gives no performance improvement. This means that I > misunderstood what this register is about, or how to change it (byte order?). > > Or the performance are constrained by something unrelated. In the blog > post cited above, the poster acheived more than 5 Gb/s before touching > MMRBC, while I am stuck at 2,7 GB/s. Any new idea welcome. > > -- > Emmanuel Dreyfus > m...@netbsd.org
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 04:40:25PM +, Taylor R Campbell wrote: > New version with some changes suggested by wiz@. Anyone has objection to this change being committed and pulled up to netbsd-7? -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: > >I also found this page that tackles the same problem on Linux: > >http://dak1n1.com/blog/7-performance-tuning-intel-10gbe It seems that page describe a slightly different model. Intel 82599 datasheet is available here: http://www.intel.fr/content/www/fr/fr/ethernet-controllers/82599-10-gbe-controller-datasheet.html No reference to MMRBC in this document, but I understand "Max Read Request Size" is the same thing. Page 765 tells us about register A8, bits 12-14 that should be set to 100. pcictl /dev/pci5 read -d 0 -f 1 0x18 tells me the value 0x00092810 I tried this command: pcictl /dev/pci5 write -d 0 -f 1 0x18 0x00094810 Further pcictl read suggests it works as the new value is returned. However it gives no performance improvement. This means that I misunderstood what this register is about, or how to change it (byte order?). Or the performance are constrained by something unrelated. In the blog post cited above, the poster acheived more than 5 Gb/s before touching MMRBC, while I am stuck at 2,7 GB/s. Any new idea welcome. -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 04:40:25PM +, Taylor R Campbell wrote: >How about the attached patch? I've been sitting on this for months. Both changes seem fine, but the board does not behave as told by Linux crowd. At 0xe6 is a nul value where we should have 0x22, and attemps to change it does not seem to have any effect. -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
Thor Lancelot Simon wrote: > MTU 9000 considered harmful. Use something that fits in 8K with the headers. > It's a minor piece of the puzzle but nonetheless, it's a piece. mtu 8192 or 8000 does not cause any improvement over mtu 9000. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org
Re: ixg(4) performances
Thor, The NetBSD TCP stack can't handle 8K payload by page-flipping the payload and prepending an mbuf for XDR/NFS/TCP/IP headers? Or is the issue the extra page-mapping for the prepended mbuf? On Tue, 8/26/14, Thor Lancelot Simon wrote: Subject: Re: ixg(4) performances To: "Emmanuel Dreyfus" Cc: tech-kern@netbsd.org Date: Tuesday, August 26, 2014, 6:56 PM [...] MTU 9000 considered harmful. Use something that fits in 8K with the headers. It's a minor piece of the puzzle but nonetheless, it's a piece. Thor
Re: ixg(4) performances
Date: Tue, 26 Aug 2014 12:44:43 -0500 From: David Young Finally, adding cfgread/cfgwrite commands to pcictl seems like a step in the wrong direction. I know that this is UNIX and we're duty-bound to give everyone enough rope, but may we reconsider our assisted-suicide policy just this one time? :-) It's certainly wrong to rely on pcictl to read and write config registers, but it's useful as a debugging tool and for driver development -- just like the rest of pcictl.
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 07:03:06PM -0700, Jonathan Stone wrote: > Thor, > > The NetBSD TCP stack can't handle 8K payload by page-flipping the payload > and prepending an mbuf for XDR/NFS/TCP/IP headers? Or is the issue the extra > page-mapping for the prepended mbuf? The issue is allocating the extra page for a milligram of data. It is almost always a lose. Better to choose the MTU so that the whole packet fits neatly in 8192 bytes. It is helpful to understand where MTU 9000 came from: SGI was trying to optimise UDP NFS performance, for NFSv2 with 8K maximum RPC size, on systems that had 16K pages. You can't fit two of that kind of NFS request in a 16K page, so you might as well allocate something a little bigger than 8K but that happens to leave your memory allocator some useful-sized chunks to hand out to other callers. I am a little hazy on the details, but I believe they ended up at MTU 9024 which is 8K + 768 + 64 (leaving a bunch of handy power-of-2 split sizes as residuals: 4096 + 2048 + 1024 + 128 + 64) which just made no sense to anyone else so everyone _else_ picked random sizes around 9000 that happened to work for their hardware. But at the end of the day, if you do not have 16K pages or are not optimizing for 8K NFSv2 requests on UDP, an MTU that fits in 8K is almost always better. Thor
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 12:17:28PM +, Emmanuel Dreyfus wrote: > Hi > > ixgb(4) has poor performances, even on latest -current. Here is the > dmesg output: > ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, > Version - 2.3.10 > ixg1: clearing prefetchable bit > ixg1: interrupting at ioapic0 pin 9 > ixg1: PCI Express Bus: Speed 2.5Gb/s Width x8 > > The interface is configued with: > ifconfig ixg1 mtu 9000 tso4 ip4csum tcp4csum-tx udp4csum-tx MTU 9000 considered harmful. Use something that fits in 8K with the headers. It's a minor piece of the puzzle but nonetheless, it's a piece. Thor
Re: ixg(4) performances
On Tue, 26 Aug 2014, David Young wrote: How well has blindly poking configuration registers worked for us in the past? Well, with the part he's using (the 82599, I think) it shouldn't be that blind. The datasheet has all the registers listed, which is the case for most of Intel's Ethernet controllers. -- Hisashi T Fujinaka - ht...@twofifty.com BSEE(6/86) + BSChem(3/95) + BAEnglish(8/95) + MSCS(8/03) + $2.50 = latte
re: ixg(4) performances
> Finally, adding cfgread/cfgwrite commands to pcictl seems like a step in > the wrong direction. I know that this is UNIX and we're duty-bound to > give everyone enough rope, but may we reconsider our assisted-suicide > policy just this one time? :-) > > How well has blindly poking configuration registers worked for us in > the past? I can think of a couple of instances where an knowledgeable > developer thought that they were writing a helpful value to a useful > register and getting a desirable result, but in the end it turned out to > be a no-op. In one case, it was an Atheros WLAN adapter where somebody > added to Linux some code that wrote to a mysterious PCI configuration > register, and then some of the *BSDs copied it. In the other case, I > think that somebody used pci_conf_write() to write a magic value to a > USB host controller register that wasn't on a 32-bit boundary. ISTR > that some incorrect value was written, instead. pciutils' "setpci" utility has exposed this for lots of systems for years. i don't see any value in keeping pcictl from being as usable as other tools, and as you say, this is unix - rope and all. .mrg.
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: > On Aug 26, 2:23pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: > -- Subject: Re: ixg(4) performances > > | On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: > | > > ftp://ftp.supermicro.com/CDR-C2_1.20_for_Intel_C2_platform/Intel/LAN/v15.5/PROXGB/DOCS/SERVER/prform10.htm#Setting_MMRBC > | > | Right, but NetBSD has no tool like Linux's setpci to tweak MMRBC, and if > | the BIOS has no setting for it, NetBSD is screwed. > | > | I see has a PCI_IOC_CFGREAD / PCI_IOC_CFGWRITE ioctl, > | does that means Linux's setpci can be easily reproduced? > > I would probably extend pcictl with cfgread and cfgwrite commands. Emmanuel, Most (all?) configuration registers are read/write. Have you read the MMRBC and found that it's improperly configured? Are you sure that you don't have to program the MMRBC at every bus bridge between the NIC and RAM? I'm not too familiar with PCI Express, so I really don't know. Have you verified the information at http://dak1n1.com/blog/7-performance-tuning-intel-10gbe with the 82599 manual? I have tried to corroborate the information both with my PCI Express book and with the 82599 manual, but I cannot make a match. PCI-X != PCI Express; maybe ixgb != ixgbe? (It sure looks like they're writing about an 82599, but maybe they don't know what they're writing about!) Finally, adding cfgread/cfgwrite commands to pcictl seems like a step in the wrong direction. I know that this is UNIX and we're duty-bound to give everyone enough rope, but may we reconsider our assisted-suicide policy just this one time? :-) How well has blindly poking configuration registers worked for us in the past? I can think of a couple of instances where an knowledgeable developer thought that they were writing a helpful value to a useful register and getting a desirable result, but in the end it turned out to be a no-op. In one case, it was an Atheros WLAN adapter where somebody added to Linux some code that wrote to a mysterious PCI configuration register, and then some of the *BSDs copied it. In the other case, I think that somebody used pci_conf_write() to write a magic value to a USB host controller register that wasn't on a 32-bit boundary. ISTR that some incorrect value was written, instead. Dave -- David Young dyo...@pobox.comUrbana, IL(217) 721-9981
Re: ixg(4) performances
Date: Tue, 26 Aug 2014 14:42:55 + From: Emmanuel Dreyfus On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: > I would probably extend pcictl with cfgread and cfgwrite commands. Sure, once it works I can do that, but a first attempt just ets EINVAL, any idea what can be wrong? ... pbcr.bus = 5; pbcr.device = 0; pbcr.function = 0; pbcr.cfgreg.reg = 0xe6b; pbcr.cfgreg.val = 0x2e; Can't do unaligned register reads/writes. If you need other than 32-bit access, you need to select subwords for reads or do R/M/W for writes. Inside the kernel, the only EINVAL is here: if (bdfr->bus > 255 || bdfr->device >= sc->sc_maxndevs || bdfr->function > 7) return EINVAL; Old kernel sources? I added a check recently for 32-bit alignment -- without which you'd hit a kassert or hardware trap shortly afterward.
Re: ixg(4) performances
Date: Tue, 26 Aug 2014 15:40:41 + From: Taylor R Campbell How about the attached patch? I've been sitting on this for months. New version with some changes suggested by wiz@. Index: usr.sbin/pcictl/pcictl.8 === RCS file: /cvsroot/src/usr.sbin/pcictl/pcictl.8,v retrieving revision 1.11 diff -p -u -r1.11 pcictl.8 --- usr.sbin/pcictl/pcictl.826 Aug 2014 16:21:15 - 1.11 +++ usr.sbin/pcictl/pcictl.826 Aug 2014 16:38:36 - @@ -33,7 +33,7 @@ .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" -.Dd February 25, 2011 +.Dd June 12, 2014 .Dt PCICTL 8 .Os .Sh NAME @@ -79,6 +79,31 @@ at the specified bus, device, and functi If the bus is not specified, it defaults to the bus number of the PCI bus specified on the command line. If the function is not specified, it defaults to 0. +.Pp +.Cm read +.Op Fl b Ar bus +.Fl d Ar device +.Op Fl f Ar function +.Ar reg +.Pp +Read the specified 32-bit aligned PCI configuration register and print +it in hexadecimal to standard output. +If the bus is not specified, it defaults to the bus number of the +PCI bus specified on the command line. +If the function is not specified, it defaults to 0. +.Pp +.Cm write +.Op Fl b Ar bus +.Fl d Ar device +.Op Fl f Ar function +.Ar reg +.Ar value +.Pp +Write the specified value to the specified 32-bit aligned PCI +configuration register. +If the bus is not specified, it defaults to the bus number of the +PCI bus specified on the command line. +If the function is not specified, it defaults to 0. .Sh FILES .Pa /dev/pci* - PCI bus device nodes Index: usr.sbin/pcictl/pcictl.c === RCS file: /cvsroot/src/usr.sbin/pcictl/pcictl.c,v retrieving revision 1.18 diff -p -u -r1.18 pcictl.c --- usr.sbin/pcictl/pcictl.c30 Aug 2011 20:08:38 - 1.18 +++ usr.sbin/pcictl/pcictl.c26 Aug 2014 16:38:36 - @@ -76,6 +76,8 @@ static intprint_numbers = 0; static voidcmd_list(int, char *[]); static voidcmd_dump(int, char *[]); +static voidcmd_read(int, char *[]); +static voidcmd_write(int, char *[]); static const struct command commands[] = { { "list", @@ -88,10 +90,21 @@ static const struct command commands[] = cmd_dump, O_RDONLY }, + { "read", + "[-b bus] -d device [-f function] reg", + cmd_read, + O_RDONLY }, + + { "write", + "[-b bus] -d device [-f function] reg value", + cmd_write, + O_WRONLY }, + { 0, 0, 0, 0 }, }; static int parse_bdf(const char *); +static u_int parse_reg(const char *); static voidscan_pci(int, int, int, void (*)(u_int, u_int, u_int)); @@ -230,6 +243,91 @@ cmd_dump(int argc, char *argv[]) scan_pci(bus, dev, func, scan_pci_dump); } +static void +cmd_read(int argc, char *argv[]) +{ + int bus, dev, func; + u_int reg; + pcireg_t value; + int ch; + + bus = pci_businfo.busno; + func = 0; + dev = -1; + + while ((ch = getopt(argc, argv, "b:d:f:")) != -1) { + switch (ch) { + case 'b': + bus = parse_bdf(optarg); + break; + case 'd': + dev = parse_bdf(optarg); + break; + case 'f': + func = parse_bdf(optarg); + break; + default: + usage(); + } + } + argv += optind; + argc -= optind; + + if (argc != 1) + usage(); + if (dev == -1) + errx(EXIT_FAILURE, "read: must specify a device number"); + reg = parse_reg(argv[0]); + if (pcibus_conf_read(pcifd, bus, dev, func, reg, &value) == -1) + err(EXIT_FAILURE, "pcibus_conf_read" + "(bus %d dev %d func %d reg %u)", bus, dev, func, reg); + if (printf("%08x\n", value) < 0) + err(EXIT_FAILURE, "printf"); +} + +static void +cmd_write(int argc, char *argv[]) +{ + int bus, dev, func; + u_int reg; + pcireg_t value; + int ch; + + bus = pci_businfo.busno; + func = 0; + dev = -1; + + while ((ch = getopt(argc, argv, "b:d:f:")) != -1) { + switch (ch) { + case 'b': + bus = parse_bdf(optarg); + break; + case 'd': + dev = parse_bdf(optarg); + break; + case 'f': + func = parse_bdf(optarg); + break; + default: + usage(); + } + } + argv += optind; + argc -= optind; + + if (argc != 2) + usage(); + if (dev == -1) +
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 11:13:50AM -0400, Christos Zoulas wrote: > I think in the example that was 0xe6. I think the .b means byte access > (I am guessing). Yes, I came to that conclusion reading pciutils sources. I discovered they also had a man page explaining that -) > I think that we are only doing word accesses, thus > we probably need to read, mask modify write the byte. I have not > verified any of that, these are guesses... Look at the pcictl source > code. I try writting at register 0xe4, but when reading again it is still 0. if (pcibus_conf_read(fd, 5, 0, 1, 0x00e4, &val) != 0) err(EX_OSERR, "pcibus_conf_read failed"); printf("reg = 0x00e4, val = 0x%08x\n", val); val = (val & 0xff00) | 0x002e; if (pcibus_conf_write(fd, 5, 0, 1, 0x00e4, val) != 0) err(EX_OSERR, "pcibus_conf_write failed"); -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
Date: Tue, 26 Aug 2014 10:25:52 -0400 From: chris...@zoulas.com (Christos Zoulas) On Aug 26, 2:23pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: -- Subject: Re: ixg(4) performances | I see has a PCI_IOC_CFGREAD / PCI_IOC_CFGWRITE ioctl, | does that means Linux's setpci can be easily reproduced? I would probably extend pcictl with cfgread and cfgwrite commands. How about the attached patch? I've been sitting on this for months. Index: usr.sbin/pcictl/pcictl.8 === RCS file: /cvsroot/src/usr.sbin/pcictl/pcictl.8,v retrieving revision 1.10 diff -p -u -r1.10 pcictl.8 --- usr.sbin/pcictl/pcictl.825 Feb 2011 21:40:48 - 1.10 +++ usr.sbin/pcictl/pcictl.826 Aug 2014 15:38:55 - @@ -33,7 +33,7 @@ .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" -.Dd February 25, 2011 +.Dd June 12, 2014 .Dt PCICTL 8 .Os .Sh NAME @@ -79,6 +79,31 @@ at the specified bus, device, and functi If the bus is not specified, it defaults to the bus number of the PCI bus specified on the command line. If the function is not specified, it defaults to 0. +.Pp +.Nm read +.Op Fl b Ar bus +.Fl d Ar device +.Op Fl f Ar function +.Ar reg +.Pp +Read the specified 32-bit aligned PCI configuration register and print +it in hexadecimal to standard output. +If the bus is not specified, it defaults to the bus number of the +PCI bus specified on the command line. +If the function is not specified, it defaults to 0. +.Pp +.Nm write +.Op Fl b Ar bus +.Fl d Ar device +.Op Fl f Ar function +.Ar reg +.Ar value +.Pp +Write the specified value to the specified 32-bit aligned PCI +configuration register. +If the bus is not specified, it defaults to the bus number of the +PCI bus specified on the command line. +If the function is not specified, it defaults to 0. .Sh FILES .Pa /dev/pci* - PCI bus device nodes Index: usr.sbin/pcictl/pcictl.c === RCS file: /cvsroot/src/usr.sbin/pcictl/pcictl.c,v retrieving revision 1.18 diff -p -u -r1.18 pcictl.c --- usr.sbin/pcictl/pcictl.c30 Aug 2011 20:08:38 - 1.18 +++ usr.sbin/pcictl/pcictl.c26 Aug 2014 15:38:55 - @@ -76,6 +76,8 @@ static intprint_numbers = 0; static voidcmd_list(int, char *[]); static voidcmd_dump(int, char *[]); +static voidcmd_read(int, char *[]); +static voidcmd_write(int, char *[]); static const struct command commands[] = { { "list", @@ -88,10 +90,21 @@ static const struct command commands[] = cmd_dump, O_RDONLY }, + { "read", + "[-b bus] -d device [-f function] reg", + cmd_read, + O_RDONLY }, + + { "write", + "[-b bus] -d device [-f function] reg value", + cmd_write, + O_WRONLY }, + { 0, 0, 0, 0 }, }; static int parse_bdf(const char *); +static u_int parse_reg(const char *); static voidscan_pci(int, int, int, void (*)(u_int, u_int, u_int)); @@ -230,6 +243,87 @@ cmd_dump(int argc, char *argv[]) scan_pci(bus, dev, func, scan_pci_dump); } +static void +cmd_read(int argc, char *argv[]) +{ + int bus, dev, func; + u_int reg; + pcireg_t value; + int ch; + + bus = pci_businfo.busno; + func = 0; + dev = -1; + + while ((ch = getopt(argc, argv, "b:d:f:")) != -1) { + switch (ch) { + case 'b': + bus = parse_bdf(optarg); + break; + case 'd': + dev = parse_bdf(optarg); + break; + case 'f': + func = parse_bdf(optarg); + break; + default: + usage(); + } + } + argv += optind; + argc -= optind; + + if (argc != 1) + usage(); + reg = parse_reg(argv[0]); + if (pcibus_conf_read(pcifd, bus, dev, func, reg, &value) == -1) + err(EXIT_FAILURE, "pcibus_conf_read" + "(bus %d dev %d func %d reg %u)", bus, dev, func, reg); + if (printf("%08x\n", value) < 0) + err(EXIT_FAILURE, "printf"); +} + +static void +cmd_write(int argc, char *argv[]) +{ + int bus, dev, func; + u_int reg; + pcireg_t value; + int ch; + + bus = pci_businfo.busno; + func = 0; + dev = -1; + + while ((ch = getopt(argc, argv, "b:d:f:")) != -1) { + switch (ch) { + case 'b': + bus = parse_bdf(optarg); + break; + case 'd': +
Re: ixg(4) performances
On Aug 26, 2:42pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: -- Subject: Re: ixg(4) performances | On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: | > I would probably extend pcictl with cfgread and cfgwrite commands. | | Sure, once it works I can do that, but a first attempt just | ets EINVAL, any idea what can be wrong? | | int fd; | struct pciio_bdf_cfgreg pbcr; | | if ((fd = open("/dev/pci5", O_RDWR, 0)) == -1) | err(EX_OSERR, "open /dev/pci5 failed"); | | pbcr.bus = 5; | pbcr.device = 0; | pbcr.function = 0; | pbcr.cfgreg.reg = 0xe6b; | pbcr.cfgreg.val = 0x2e; I think in the example that was 0xe6. I think the .b means byte access (I am guessing). I think that we are only doing word accesses, thus we probably need to read, mask modify write the byte. I have not verified any of that, these are guesses... Look at the pcictl source code. | | if (ioctl(fd, PCI_IOC_BDF_CFGWRITE, &pbcr) == -1) | err(EX_OSERR, "ioctl failed"); | | Inside the kernel, the only EINVAL is here: | if (bdfr->bus > 255 || bdfr->device >= sc->sc_maxndevs || | bdfr->function > 7) | return EINVAL; | | -- | Emmanuel Dreyfus | m...@netbsd.org -- End of excerpt from Emmanuel Dreyfus
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: > I would probably extend pcictl with cfgread and cfgwrite commands. Sure, once it works I can do that, but a first attempt just ets EINVAL, any idea what can be wrong? int fd; struct pciio_bdf_cfgreg pbcr; if ((fd = open("/dev/pci5", O_RDWR, 0)) == -1) err(EX_OSERR, "open /dev/pci5 failed"); pbcr.bus = 5; pbcr.device = 0; pbcr.function = 0; pbcr.cfgreg.reg = 0xe6b; pbcr.cfgreg.val = 0x2e; if (ioctl(fd, PCI_IOC_BDF_CFGWRITE, &pbcr) == -1) err(EX_OSERR, "ioctl failed"); Inside the kernel, the only EINVAL is here: if (bdfr->bus > 255 || bdfr->device >= sc->sc_maxndevs || bdfr->function > 7) return EINVAL; -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
On Aug 26, 2:23pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: -- Subject: Re: ixg(4) performances | On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: | > ftp://ftp.supermicro.com/CDR-C2_1.20_for_Intel_C2_platform/Intel/LAN/v15.5/PROXGB/DOCS/SERVER/prform10.htm#Setting_MMRBC | | Right, but NetBSD has no tool like Linux's setpci to tweak MMRBC, and if | the BIOS has no setting for it, NetBSD is screwed. | | I see has a PCI_IOC_CFGREAD / PCI_IOC_CFGWRITE ioctl, | does that means Linux's setpci can be easily reproduced? I would probably extend pcictl with cfgread and cfgwrite commands. christos
Re: ixg(4) performances
On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: > ftp://ftp.supermicro.com/CDR-C2_1.20_for_Intel_C2_platform/Intel/LAN/v15.5/PROXGB/DOCS/SERVER/prform10.htm#Setting_MMRBC Right, but NetBSD has no tool like Linux's setpci to tweak MMRBC, and if the BIOS has no setting for it, NetBSD is screwed. I see has a PCI_IOC_CFGREAD / PCI_IOC_CFGWRITE ioctl, does that means Linux's setpci can be easily reproduced? -- Emmanuel Dreyfus m...@netbsd.org
Re: ixg(4) performances
In article <20140826121728.gl23...@homeworld.netbsd.org>, Emmanuel Dreyfus wrote: >Hi > >ixgb(4) has poor performances, even on latest -current. Here is the >dmesg output: >ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network >Driver, Version - 2.3.10 >ixg1: clearing prefetchable bit >ixg1: interrupting at ioapic0 pin 9 >ixg1: PCI Express Bus: Speed 2.5Gb/s Width x8 > >The interface is configued with: >ifconfig ixg1 mtu 9000 tso4 ip4csum tcp4csum-tx udp4csum-tx > >And sysctl: >kern.sbmax = 67108864 >kern.somaxkva = 67108864 >net.inet.udp.sendspace = 2097152 >net.inet.udp.recvspace = 2097152 >net.inet.tcp.sendspace = 2097152 >net.inet.tcp.recvspace = 2097152 >net.inet.tcp.recvbuf_auto = 0 >net.inet.tcp.sendbuf_auto = 0 > >netperfs shows a maximum performance of 2.3 Gb/s. That let me with >the feeling that only a PCI lane is used. Is it possible? > >I also found this page that tackles the same problem on Linux: >http://dak1n1.com/blog/7-performance-tuning-intel-10gbe > >They tweak the PCI MMRBC. Anyone has an idea of how it could be >done on NetBSD? I thought about borrowing code from src/sys/dec/pci/if_dge.c >but I am not sure what pci_conf_read/pci_conf_write commands should be used. > >Any other idea on how to improve performance? ftp://ftp.supermicro.com/CDR-C2_1.20_for_Intel_C2_platform/Intel/LAN/v15.5/PROXGB/DOCS/SERVER/prform10.htm#Setting_MMRBC
ixg(4) performances
Hi ixgb(4) has poor performances, even on latest -current. Here is the dmesg output: ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10 ixg1: clearing prefetchable bit ixg1: interrupting at ioapic0 pin 9 ixg1: PCI Express Bus: Speed 2.5Gb/s Width x8 The interface is configued with: ifconfig ixg1 mtu 9000 tso4 ip4csum tcp4csum-tx udp4csum-tx And sysctl: kern.sbmax = 67108864 kern.somaxkva = 67108864 net.inet.udp.sendspace = 2097152 net.inet.udp.recvspace = 2097152 net.inet.tcp.sendspace = 2097152 net.inet.tcp.recvspace = 2097152 net.inet.tcp.recvbuf_auto = 0 net.inet.tcp.sendbuf_auto = 0 netperfs shows a maximum performance of 2.3 Gb/s. That let me with the feeling that only a PCI lane is used. Is it possible? I also found this page that tackles the same problem on Linux: http://dak1n1.com/blog/7-performance-tuning-intel-10gbe They tweak the PCI MMRBC. Anyone has an idea of how it could be done on NetBSD? I thought about borrowing code from src/sys/dec/pci/if_dge.c but I am not sure what pci_conf_read/pci_conf_write commands should be used. Any other idea on how to improve performance? -- Emmanuel Dreyfus m...@netbsd.org