Re: FW: ixg(4) performances

2014-10-01 Thread David Laight
On Sun, Aug 31, 2014 at 12:07:38PM -0400, Terry Moore wrote: > > This is not 2.5G Transfers per second. PCIe talks about transactions rather > than transfers; one transaction requires either 12 bytes (for 32-bit > systems) or 16 bytes (for 64-bit systems) of overhead at the transaction > layer, pl

Re: ixg(4) performances

2014-09-03 Thread Masanobu SAITOH
On 2014/09/04 11:24, Masanobu SAITOH wrote: On 2014/09/04 0:40, Emmanuel Dreyfus wrote: On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: NetBSD 6.1 says: vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 function 0 not configured In src/sys/dev/pci/ixgbe

Re: ixg(4) performances

2014-09-03 Thread Masanobu SAITOH
On 2014/09/04 0:40, Emmanuel Dreyfus wrote: On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: NetBSD 6.1 says: vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 function 0 not configured In src/sys/dev/pci/ixgbe/ we know about producct Id 0x1529 and 0x152A

RE: FW: ixg(4) performances

2014-09-03 Thread Terry Moore
>From Emmanuel Dreyfus > You are right; > # pcictl /dev/pci5 read -d 0 -f 1 0xa8 > 00092810 > # pcictl /dev/pci5 write -d 0 -f 1 0xa8 0x00094810 > # pcictl /dev/pci5 read -d 0 -f 1 0xa8 > 4810 That's reassuring. The dump confirms that we're looking at the right registers, thank you. As I read

RE: FW: ixg(4) performances

2014-09-03 Thread Terry Moore
> -Original Message- > From: Hisashi T Fujinaka [mailto:ht...@twofifty.com] > Sent: Sunday, August 31, 2014 12:39 > To: Terry Moore > Cc: tech-kern@netbsd.org > Subject: RE: FW: ixg(4) performances > > I may be wrong in the transactions/transfers. However, I th

Re: ixg(4) performances

2014-09-03 Thread Emmanuel Dreyfus
On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: > NetBSD 6.1 says: > vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 > function 0 not configured In src/sys/dev/pci/ixgbe/ we know about producct Id 0x1529 and 0x152A but not 0x1528. But this can probably be

Re: ixg(4) performances

2014-09-03 Thread Bert Kiers
On Wed, Sep 03, 2014 at 04:11:29PM +0200, Bert Kiers wrote: > NetBSD 6.1 says: > > vendor 0x8086 product 0x1528 (ethernet network, revision 0x01) at pci1 dev 0 > function 0 not configured > > Complete messages: http://netbsd.itsx.net/hw/x9drw.dmesg > > NetBSD current from today also does not co

Re: ixg(4) performances

2014-09-03 Thread Bert Kiers
On Sat, Aug 30, 2014 at 10:24:52AM +0100, Justin Cormack wrote: > On Sat, Aug 30, 2014 at 8:22 AM, Thor Lancelot Simon wrote: > > On Fri, Aug 29, 2014 at 12:22:31PM -0400, Terry Moore wrote: > >> > >> Is the ixg in an expansion slot or integrated onto the main board? > > > > If you know where to g

Re: FW: ixg(4) performances

2014-08-31 Thread Masanobu SAITOH
Hi, Emmanuel. On 2014/09/01 11:10, Emmanuel Dreyfus wrote: > Terry Moore wrote: > >> Since you did a dword read, the extra 0x9 is the device status register. >> This makes me suspicious as the device status register is claiming that you >> have "unsupported request detected)" [bit 3] and "co

Re: FW: ixg(4) performances

2014-08-31 Thread Emmanuel Dreyfus
Terry Moore wrote: > Since you did a dword read, the extra 0x9 is the device status register. > This makes me suspicious as the device status register is claiming that you > have "unsupported request detected)" [bit 3] and "correctable error > detected" [bit 0]. Further, this register is RW1

RE: FW: ixg(4) performances

2014-08-31 Thread Hisashi T Fujinaka
Oh, and to answer the actual first, relevant question, I can try finding out if we (day job, 82599) can do line rate at 2.5GT/s. I think we can get a lot closer than you're getting but we don't test with NetBSD. -- Hisashi T Fujinaka - ht...@twofifty.com BSEE + BSChem + BAEnglish + MSCS + $2.50 =

RE: FW: ixg(4) performances

2014-08-31 Thread Hisashi T Fujinaka
rate is not used when talking about the max data transfer rate. On Sun, 31 Aug 2014, Terry Moore wrote: -Original Message- From: Hisashi T Fujinaka [mailto:ht...@twofifty.com] Sent: Saturday, August 30, 2014 21:29 To: Terry Moore Cc: tech-kern@netbsd.org Subject: Re: FW: ixg(4) perform

RE: FW: ixg(4) performances

2014-08-31 Thread Terry Moore
> -Original Message- > From: Hisashi T Fujinaka [mailto:ht...@twofifty.com] > Sent: Saturday, August 30, 2014 21:29 > To: Terry Moore > Cc: tech-kern@netbsd.org > Subject: Re: FW: ixg(4) performances > > Doesn't anyone read my posts or, more important,

Re: FW: ixg(4) performances

2014-08-30 Thread Hisashi T Fujinaka
s' Subject: RE: ixg(4) performances But it's running at gen1. I strongly suspect that the benchmark case was gen2 (since the ixg is capable of it). gen1 vs gen2 is 2.5 Gb.s vs 5 Gb/s? Yes. Actually, 2.5Gbps is symbol rate -- it's 8/10 encoded, so one lane is really 2Gbps. S

FW: ixg(4) performances

2014-08-30 Thread Terry Moore
Forgot to cc the list. -Original Message- From: Terry Moore [mailto:t...@mcci.com] Sent: Friday, August 29, 2014 15:13 To: 'Emmanuel Dreyfus' Subject: RE: ixg(4) performances > > But it's running at gen1. I strongly suspect that the benchmark case > > was

Re: ixg(4) performances

2014-08-30 Thread Emmanuel Dreyfus
Matthias Drochner wrote: > > Link Capabilities Ragister (0xAC): 0x00027482 > > bits 3:0 Supprted Link speed: 0010 = 5 GbE and 2.5 GbE speed > > supported bits 9:4 Max link width: 001000 = x4 > > Wrong -- this means x8. > > > bits 14:12 L0s exit lattency: 101 = 1 µs - 2 µs > > bits 17:15 L1

Re: ixg(4) performances

2014-08-30 Thread Matthias Drochner
On Fri, 29 Aug 2014 15:51:14 + Emmanuel Dreyfus wrote: > I found this, but the result does not make sense: negociated > max ... > > Link Capabilities Ragister (0xAC): 0x00027482 > bits 3:0 Supprted Link speed: 0010 = 5 GbE and 2.5 GbE speed > supported bits 9:4 Max link width: 001000 = x

Re: ixg(4) performances

2014-08-30 Thread Justin Cormack
On Sat, Aug 30, 2014 at 8:22 AM, Thor Lancelot Simon wrote: > On Fri, Aug 29, 2014 at 12:22:31PM -0400, Terry Moore wrote: >> >> Is the ixg in an expansion slot or integrated onto the main board? > > If you know where to get a mainboard with an integrated ixg, I wouldn't > mind hearing about it.

Re: ixg(4) performances

2014-08-30 Thread Thor Lancelot Simon
On Fri, Aug 29, 2014 at 12:22:31PM -0400, Terry Moore wrote: > > Is the ixg in an expansion slot or integrated onto the main board? If you know where to get a mainboard with an integrated ixg, I wouldn't mind hearing about it. Thor

Re: ixg(4) performances

2014-08-29 Thread Hisashi T Fujinaka
On Fri, 29 Aug 2014, Emmanuel Dreyfus wrote: Terry Moore wrote: But it's running at gen1. I strongly suspect that the benchmark case was gen2 (since the ixg is capable of it). gen1 vs gen2 is 2.5 Gb.s bs 5 Gb/s? Gen 1 is capable of only 2.5GT/s (gigatransfers per second). Gen 2 is capable

Re: ixg(4) performances

2014-08-29 Thread Emmanuel Dreyfus
Terry Moore wrote: > But it's running at gen1. I strongly suspect that the benchmark case was > gen2 (since the ixg is capable of it). gen1 vs gen2 is 2.5 Gb.s bs 5 Gb/s? > Is the ixg in an expansion slot or integrated onto the main board? In a slot. -- Emmanuel Dreyfus http://hcpnet.free.fr

RE: ixg(4) performances

2014-08-29 Thread Terry Moore
> On Friday, August 29, 2014 11:51, Emmanuel Dreyfus wrote: > > On Fri, Aug 29, 2014 at 08:48:51AM -0400, Terry Moore wrote: > > Still, you should check whether you have the right number of the right > > generation of PCIe lanes connected to the ixg. > > I found this, but the result does not make

Re: ixg(4) performances

2014-08-29 Thread Emmanuel Dreyfus
On Fri, Aug 29, 2014 at 08:48:51AM -0400, Terry Moore wrote: > Still, you should check whether you have the right number of the right > generation of PCIe lanes connected to the ixg. I found this, but the result does not make sense: negociated > max ... Link Capabilities Ragister (0xAC): 0x00027

RE: ixg(4) performances

2014-08-29 Thread Terry Moore
> -Original Message- > From: tech-kern-ow...@netbsd.org [mailto:tech-kern-ow...@netbsd.org] On > Behalf Of Emmanuel Dreyfus > Sent: Thursday, August 28, 2014 23:55 > To: Terry Moore; 'Christos Zoulas' > Cc: tech-kern@netbsd.org > Subject: Re: ixg(4) perfo

Re: ixg(4) performances

2014-08-28 Thread Emmanuel Dreyfus
Terry Moore wrote: > There are several possibilities, all revolving about differences between the > blog poster's base system and yorus. Do I have a way to investigate for appropriate PCI setup? Here is what dmesg says about it: pci0 at mainbus0 bus 0: configuration mode 1 pci0: i/o space, memo

Re: ixg(4) performances

2014-08-28 Thread Emmanuel Dreyfus
On Thu, Aug 28, 2014 at 08:37:06AM -0700, Hisashi T Fujinaka wrote: > Isn't your PCIe slot constrained? I thought I remembered that you're > only getting 2.5GT/s and I forget what test you're running. I use netperf, and I now get 2.7 Gb/s. -- Emmanuel Dreyfus m...@netbsd.org

Re: ixg(4) performances

2014-08-28 Thread Hisashi T Fujinaka
On Thu, 28 Aug 2014, Emmanuel Dreyfus wrote: On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: I also found this page that tackles the same problem on Linux: http://dak1n1.com/blog/7-performance-tuning-intel-10gbe It seems that page describe a slightly different model. Intel 82

Re: ixg(4) performances

2014-08-28 Thread Christos Zoulas
In article <20140828072832.gi8...@homeworld.netbsd.org>, Emmanuel Dreyfus wrote: >On Tue, Aug 26, 2014 at 04:40:25PM +, Taylor R Campbell wrote: >> New version with some changes suggested by wiz@. > >Anyone has objection to this change being committed and pulled up to >netbsd-7? > Not me. c

RE: ixg(4) performances

2014-08-28 Thread Terry Moore
> Or the performance are constrained by something unrelated. In the blog > post cited above, the poster acheived more than 5 Gb/s before touching > MMRBC, while I am stuck at 2,7 GB/s. Any new idea welcome. The blog post refers to PCI-X, I'm more familiar with PCIe, but the concepts are similar.

Re: ixg(4) performances

2014-08-28 Thread Stephan
What is your test setup? Do you have 2 identical boxes? Does it perform better e.g. on Linux or FreeBSD? If so, you could check how the config registers get set by that particular OS. 2014-08-28 9:26 GMT+02:00 Emmanuel Dreyfus : > On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: >

Re: ixg(4) performances

2014-08-28 Thread Emmanuel Dreyfus
On Tue, Aug 26, 2014 at 04:40:25PM +, Taylor R Campbell wrote: > New version with some changes suggested by wiz@. Anyone has objection to this change being committed and pulled up to netbsd-7? -- Emmanuel Dreyfus m...@netbsd.org

Re: ixg(4) performances

2014-08-28 Thread Emmanuel Dreyfus
On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: > >I also found this page that tackles the same problem on Linux: > >http://dak1n1.com/blog/7-performance-tuning-intel-10gbe It seems that page describe a slightly different model. Intel 82599 datasheet is available here: http://www.

Re: ixg(4) performances

2014-08-27 Thread Emmanuel Dreyfus
On Tue, Aug 26, 2014 at 04:40:25PM +, Taylor R Campbell wrote: >How about the attached patch? I've been sitting on this for months. Both changes seem fine, but the board does not behave as told by Linux crowd. At 0xe6 is a nul value where we should have 0x22, and attemps to change it does

Re: ixg(4) performances

2014-08-26 Thread Emmanuel Dreyfus
Thor Lancelot Simon wrote: > MTU 9000 considered harmful. Use something that fits in 8K with the headers. > It's a minor piece of the puzzle but nonetheless, it's a piece. mtu 8192 or 8000 does not cause any improvement over mtu 9000. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd

Re: ixg(4) performances

2014-08-26 Thread Jonathan Stone
bject: Re: ixg(4) performances To: "Emmanuel Dreyfus" Cc: tech-kern@netbsd.org Date: Tuesday, August 26, 2014, 6:56 PM [...] MTU 9000 considered harmful.  Use something that fits in 8K with the headers. It's a minor piece of the puzzle but nonetheless, it's a piece. Thor

Re: ixg(4) performances

2014-08-26 Thread Taylor R Campbell
Date: Tue, 26 Aug 2014 12:44:43 -0500 From: David Young Finally, adding cfgread/cfgwrite commands to pcictl seems like a step in the wrong direction. I know that this is UNIX and we're duty-bound to give everyone enough rope, but may we reconsider our assisted-suicide policy ju

Re: ixg(4) performances

2014-08-26 Thread Thor Lancelot Simon
On Tue, Aug 26, 2014 at 07:03:06PM -0700, Jonathan Stone wrote: > Thor, > > The NetBSD TCP stack can't handle 8K payload by page-flipping the payload > and prepending an mbuf for XDR/NFS/TCP/IP headers? Or is the issue the extra > page-mapping for the prepended mbuf? The issue is allocating th

Re: ixg(4) performances

2014-08-26 Thread Thor Lancelot Simon
On Tue, Aug 26, 2014 at 12:17:28PM +, Emmanuel Dreyfus wrote: > Hi > > ixgb(4) has poor performances, even on latest -current. Here is the > dmesg output: > ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, > Version - 2.3.10 > ixg1: clearing prefetchable bit > ixg

Re: ixg(4) performances

2014-08-26 Thread Hisashi T Fujinaka
On Tue, 26 Aug 2014, David Young wrote: How well has blindly poking configuration registers worked for us in the past? Well, with the part he's using (the 82599, I think) it shouldn't be that blind. The datasheet has all the registers listed, which is the case for most of Intel's Ethernet cont

re: ixg(4) performances

2014-08-26 Thread matthew green
> Finally, adding cfgread/cfgwrite commands to pcictl seems like a step in > the wrong direction. I know that this is UNIX and we're duty-bound to > give everyone enough rope, but may we reconsider our assisted-suicide > policy just this one time? :-) > > How well has blindly poking configuratio

Re: ixg(4) performances

2014-08-26 Thread David Young
On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: > On Aug 26, 2:23pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: > -- Subject: Re: ixg(4) performances > > | On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: > | > > ftp://f

Re: ixg(4) performances

2014-08-26 Thread Taylor R Campbell
Date: Tue, 26 Aug 2014 14:42:55 + From: Emmanuel Dreyfus On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: > I would probably extend pcictl with cfgread and cfgwrite commands. Sure, once it works I can do that, but a first attempt just ets EINVAL, any idea wh

Re: ixg(4) performances

2014-08-26 Thread Taylor R Campbell
Date: Tue, 26 Aug 2014 15:40:41 + From: Taylor R Campbell How about the attached patch? I've been sitting on this for months. New version with some changes suggested by wiz@. Index: usr.sbin/pcictl/pcictl.8 === RCS fil

Re: ixg(4) performances

2014-08-26 Thread Emmanuel Dreyfus
On Tue, Aug 26, 2014 at 11:13:50AM -0400, Christos Zoulas wrote: > I think in the example that was 0xe6. I think the .b means byte access > (I am guessing). Yes, I came to that conclusion reading pciutils sources. I discovered they also had a man page explaining that -) > I think that we are onl

Re: ixg(4) performances

2014-08-26 Thread Taylor R Campbell
Date: Tue, 26 Aug 2014 10:25:52 -0400 From: chris...@zoulas.com (Christos Zoulas) On Aug 26, 2:23pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: -- Subject: Re: ixg(4) performances | I see has a PCI_IOC_CFGREAD / PCI_IOC_CFGWRITE ioctl, | does that means Linux's setpci c

Re: ixg(4) performances

2014-08-26 Thread Christos Zoulas
On Aug 26, 2:42pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: -- Subject: Re: ixg(4) performances | On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: | > I would probably extend pcictl with cfgread and cfgwrite commands. | | Sure, once it works I can do that, but a first atte

Re: ixg(4) performances

2014-08-26 Thread Emmanuel Dreyfus
On Tue, Aug 26, 2014 at 10:25:52AM -0400, Christos Zoulas wrote: > I would probably extend pcictl with cfgread and cfgwrite commands. Sure, once it works I can do that, but a first attempt just ets EINVAL, any idea what can be wrong? int fd; struct pciio_bdf_cfgreg pbcr;

Re: ixg(4) performances

2014-08-26 Thread Christos Zoulas
On Aug 26, 2:23pm, m...@netbsd.org (Emmanuel Dreyfus) wrote: -- Subject: Re: ixg(4) performances | On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: | > ftp://ftp.supermicro.com/CDR-C2_1.20_for_Intel_C2_platform/Intel/LAN/v15.5/PROXGB/DOCS/SERVER/prform10.htm#Setting_MM

Re: ixg(4) performances

2014-08-26 Thread Emmanuel Dreyfus
On Tue, Aug 26, 2014 at 12:57:37PM +, Christos Zoulas wrote: > ftp://ftp.supermicro.com/CDR-C2_1.20_for_Intel_C2_platform/Intel/LAN/v15.5/PROXGB/DOCS/SERVER/prform10.htm#Setting_MMRBC Right, but NetBSD has no tool like Linux's setpci to tweak MMRBC, and if the BIOS has no setting for it, NetBS

Re: ixg(4) performances

2014-08-26 Thread Christos Zoulas
In article <20140826121728.gl23...@homeworld.netbsd.org>, Emmanuel Dreyfus wrote: >Hi > >ixgb(4) has poor performances, even on latest -current. Here is the >dmesg output: >ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network >Driver, Version - 2.3.10 >ixg1: clearing prefetchable

ixg(4) performances

2014-08-26 Thread Emmanuel Dreyfus
Hi ixgb(4) has poor performances, even on latest -current. Here is the dmesg output: ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10 ixg1: clearing prefetchable bit ixg1: interrupting at ioapic0 pin 9 ixg1: PCI Express Bus: Speed 2.5Gb/s Width x8 Th