Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Bruce Evans]

On Tue, 24 Jan 2017, Sean Bruno wrote:


On 01/24/17 08:27, Olivier Cochard-Labb?? wrote:

On Tue, Jan 24, 2017 at 3:17 PM, Sean Bruno > wrote:

Did you increase the number of rx/tx rings to 8 and the number of
descriptors to 4k in your tests or just the defaults?

Tuning are same as described in my previous email (rxd|txd=2048, rx|tx
process_limit=-1, max_interrupt_rate=16000).
[root@apu2]~# sysctl hw.igb.
hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1
hw.igb.num_queues: 0
hw.igb.header_split: 0
hw.igb.max_interrupt_rate: 16000
hw.igb.enable_msix: 1
hw.igb.enable_aim: 1
hw.igb.txd: 2048
hw.igb.rxd: 2048


Oh, I think you missed my note on these.  In order to adjust txd/rxd you
need to tweak the iflib version of these numbers.  nrxds/ntxds should be
adjust upwards to your value of 2048.  nrxqs/ntxqs should be adjust
upwards to 8, I think, so you can test equivalent settings to the legacy
driver.

Specifically, you may want to adjust these:

dev.em.0.iflib.override_nrxds: 0
dev.em.0.iflib.override_ntxds: 0

dev.em.0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntxqs: 0


That is painful.

My hack to increase the ifq length also no longer works:

X Index: if_em.c
X ===
X --- if_em.c   (revision 312696)
X +++ if_em.c   (working copy)
X @@ -1,3 +1,5 @@
X +int em_qlenadj = -1;
X +

-1 gives a null adjustment; 0 gives a default (very large ifq), and other
values give a non-null adustment.

X  /*-
X   * Copyright (c) 2016 Matt Macy 
X   * All rights reserved.
X @@ -2488,7 +2490,10 @@
X 
X  	/* Single Queue */

X  if (adapter->tx_num_queues == 1) {
X -   if_setsendqlen(ifp, scctx->isc_ntxd[0] - 1);
X +   if (em_qlenadj == 0)
X + em_qlenadj = imax(2 * tick, 0) * 15 / 10;
X + // lem_qlenadj = imax(2 * tick, 0) * 42 / 100;
X +   if_setsendqlen(ifp, scctx->isc_ntxd[0] + em_qlenadj);
X if_setsendqready(ifp);
X   }
X

I don't want larger hardware queues, but sometimes want larger software
queues.  ifq's used to give them.  The if_setsenqlen() call is still there.
but no longer gives them.

The large queues are needed for backet blasting benchmarks since select()
doesn't work for udp sockets, so if the queues fill up then the benchmarks
must busy-wait or sleep waiting for them to drain, and timeout granularity
tends to prevent short sleeps from working so the queues run dry while
sleeping unless the queues are very large.

Bruce___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/24/17 08:27, Olivier Cochard-Labbé wrote:
> On Tue, Jan 24, 2017 at 3:17 PM, Sean Bruno  > wrote:
> 
> 
> 
> Did you increase the number of rx/tx rings to 8 and the number of
> descriptors to 4k in your tests or just the defaults?
> 
> 
> Tuning are same as described in my previous email (rxd|txd=2048, rx|tx
> process_limit=-1, max_interrupt_rate=16000).
> [root@apu2]~# sysctl hw.igb.
> hw.igb.tx_process_limit: -1
> hw.igb.rx_process_limit: -1
> hw.igb.num_queues: 0
> hw.igb.header_split: 0
> hw.igb.max_interrupt_rate: 16000
> hw.igb.enable_msix: 1
> hw.igb.enable_aim: 1
> hw.igb.txd: 2048
> hw.igb.rxd: 2048
> 
> 

Oh, I think you missed my note on these.  In order to adjust txd/rxd you
need to tweak the iflib version of these numbers.  nrxds/ntxds should be
adjust upwards to your value of 2048.  nrxqs/ntxqs should be adjust
upwards to 8, I think, so you can test equivalent settings to the legacy
driver.

Specifically, you may want to adjust these:

dev.em.0.iflib.override_nrxds: 0
dev.em.0.iflib.override_ntxds: 0

dev.em.0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntxqs: 0

sean

> But I've did a new benchs with default setting, and the performance drop
> is now about -25% :
> 
> x head r311848 packets-per-second (default settings)
> + head r311849 packets-per-second (default settings)
> +--+
> |+ |
> |+   x |
> |+   xx|
> |++  xx|
> |A||
> |A||
> +--+
> N   Min   MaxMedian   AvgStddev
> x   5618711621135  619930.5  619840.8 951.83787
> +   5467389468740467778  467864.8 550.40322
> Difference at 95.0% confidence
> -151976 +/- 1133.9
> -24.5186% +/- 0.150581%
> (Student's t, pooled s = 777.476)
> 



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Tue, Jan 24, 2017 at 3:17 PM, Sean Bruno  wrote:

>
>
> Did you increase the number of rx/tx rings to 8 and the number of
> descriptors to 4k in your tests or just the defaults?
>

Tuning are same as described in my previous email (rxd|txd=2048, rx|tx
process_limit=-1, max_interrupt_rate=16000).
[root@apu2]~# sysctl hw.igb.
hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1
hw.igb.num_queues: 0
hw.igb.header_split: 0
hw.igb.max_interrupt_rate: 16000
hw.igb.enable_msix: 1
hw.igb.enable_aim: 1
hw.igb.txd: 2048
hw.igb.rxd: 2048


But I've did a new benchs with default setting, and the performance drop is
now about -25% :

x head r311848 packets-per-second (default settings)
+ head r311849 packets-per-second (default settings)
+--+
|+ |
|+   x |
|+   xx|
|++  xx|
|A||
|A||
+--+
N   Min   MaxMedian   AvgStddev
x   5618711621135  619930.5  619840.8 951.83787
+   5467389468740467778  467864.8 550.40322
Difference at 95.0% confidence
-151976 +/- 1133.9
-24.5186% +/- 0.150581%
(Student's t, pooled s = 777.476)
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/23/17 23:31, Olivier Cochard-Labbé wrote:
> On Tue, Jan 24, 2017 at 2:40 AM, Sean Bruno  > wrote:
> 
> 
> 
> Which set of configs from your test suite are you using for this?
> Specifically, what packet size are you slamming across?
> 
> https://github.com/ocochard/netbenches/tree/master/pktgen.configs
> 
> 
> 
> ​Because I'm in the point of view of a Telco, I'm measuring the «worst»
> case, this mean with the smallest frame size.
> Here is the exact pkt-gen command line I'm using:
> - 60 byte Ethernet frame size (excluding the 4 CRC bytes)
> - 2000 UDP flows (20 IP sources * 100 IP destinations)
> 
> pkt-gen -U -i igb2 -f tx -n 8000 -l 60 -d 198.19.10.1:2000-198.19.10.20 
> -D 00:0d:b9:41:ca:3d -s 198.18.10.1:2000-198.18.10.100 -w 4
> 
> ​Option -U is available on a patched netmap version [1]: It fix the
> checksum calculation when using source/destination IP range on NIC that
> didn't enable HW CHKSUM in netmap mode and IPv6 support.
> 
> [1]
> https://github.com/ocochard/BSDRP/blob/master/BSDRPcur/patches/freebsd.pkt-gen.ae-ipv6.patch
> 


Did you increase the number of rx/tx rings to 8 and the number of
descriptors to 4k in your tests or just the defaults?

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Tue, Jan 24, 2017 at 2:40 AM, Sean Bruno  wrote:

>
>
> Which set of configs from your test suite are you using for this?
> Specifically, what packet size are you slamming across?
>
> https://github.com/ocochard/netbenches/tree/master/pktgen.configs
>

​Because I'm in the point of view of a Telco, I'm measuring the «worst»
case, this mean with the smallest frame size.
Here is the exact pkt-gen command line I'm using:
- 60 byte Ethernet frame size (excluding the 4 CRC bytes)
- 2000 UDP flows (20 IP sources * 100 IP destinations)

pkt-gen -U -i igb2 -f tx -n 8000 -l 60 -d
198.19.10.1:2000-198.19.10.20 -D 00:0d:b9:41:ca:3d -s
198.18.10.1:2000-198.18.10.100 -w 4

​Option -U is available on a patched netmap version [1]: It fix the
checksum calculation when using source/destination IP range on NIC that
didn't enable HW CHKSUM in netmap mode and IPv6 support.

[1]
https://github.com/ocochard/BSDRP/blob/master/BSDRPcur/patches/freebsd.pkt-gen.ae-ipv6.patch
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/23/17 08:39, Olivier Cochard-Labbé wrote:
> 
> On Thu, Jan 12, 2017 at 1:54 AM, Matthew Macy  > wrote:
> 
>  >  A flame graph for the core cycle count and a flame graph with
> cache miss stats from pmc would be a great start.
>  >
>  >
>  > ​I didn't know the exact event name to use for cache miss stats,
> but here are the flame graphs for CPU_CLK_UNHALTED_CORE:
>  > http://dev.bsdrp.net/netgate.r311848.CPU_CLK_UNHALTED_CORE.svg
> 
>  > http://dev.bsdrp.net/netgate.r311849.CPU_CLK_UNHALTED_CORE.svg
> 
> 
> Thanks. Having twice as many txqs would definitely help. It's also
> clear that there may be some sort of peformance issue in
> iflib_txq_drain. Although it could just be non-stop cache misses on
> the packet headers.
> 
> 
> ​Any news about the performance issue in iflib_txq_drain ?
> 
> On a different hardware (PC Engine APU2), I've got -20% performance drop:
> 
> x head r311848: packets per second
> + head r311849: packets per second
> +--+
> | ++  x|
> |+++ x xx x|
> | |_A_||
> ||A|   |
> +--+
> N   Min   MaxMedian   AvgStddev
> x   5580021588650585676  585406.1 3550.8673
> +   5463865467599465428  465638.6 1437.9347
> Difference at 95.0% confidence
> -119768 +/- 3950.78
> -20.4589% +/- 0.558328%
> (Student's t, pooled s = 2708.9)
> ​
>  
> ​Because it's an AMD processor I didn't found the pmc equivalent of
> CPU_CLK_UNHALTED_CORE, then I've used BU_CPU_CLK_UNHALTED but I've no
> idea if it's the good one.
> 
> http://dev.bsdrp.net/apu2.r311848.BU_CPU_CLK_UNHALTED.svg
> http://dev.bsdrp.net/apu2.r311849.BU_CPU_CLK_UNHALTED.svg
> ​
> ​Thanks​
> 
> 


Olivier:

Which set of configs from your test suite are you using for this?
Specifically, what packet size are you slamming across?

https://github.com/ocochard/netbenches/tree/master/pktgen.configs

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Thu, Jan 12, 2017 at 1:54 AM, Matthew Macy  wrote:

>  >  A flame graph for the core cycle count and a flame graph with cache
> miss stats from pmc would be a great start.
>  >
>  >
>  > ​I didn't know the exact event name to use for cache miss stats, but
> here are the flame graphs for CPU_CLK_UNHALTED_CORE:
>  > http://dev.bsdrp.net/netgate.r311848.CPU_CLK_UNHALTED_CORE.svg
>  > http://dev.bsdrp.net/netgate.r311849.CPU_CLK_UNHALTED_CORE.svg
>
> Thanks. Having twice as many txqs would definitely help. It's also clear
> that there may be some sort of peformance issue in iflib_txq_drain.
> Although it could just be non-stop cache misses on the packet headers.
>
>
> ​Any news about the performance issue in iflib_txq_drain ?

On a different hardware (PC Engine APU2), I've got -20% performance drop:

x head r311848: packets per second
+ head r311849: packets per second
+--+
| ++  x|
|+++ x xx x|
| |_A_||
||A|   |
+--+
N   Min   MaxMedian   AvgStddev
x   5580021588650585676  585406.1 3550.8673
+   5463865467599465428  465638.6 1437.9347
Difference at 95.0% confidence
-119768 +/- 3950.78
-20.4589% +/- 0.558328%
(Student's t, pooled s = 2708.9)
​

​Because it's an AMD processor I didn't found the pmc equivalent of
CPU_CLK_UNHALTED_CORE, then I've used BU_CPU_CLK_UNHALTED but I've no idea
if it's the good one.

http://dev.bsdrp.net/apu2.r311848.BU_CPU_CLK_UNHALTED.svg
http://dev.bsdrp.net/apu2.r311849.BU_CPU_CLK_UNHALTED.svg
​
​Thanks​
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[O. Hartmann]

On Wed, 18 Jan 2017 07:59:17 -0700
Sean Bruno  wrote:

> On 01/18/17 07:41, Sean Bruno wrote:
> > 
> > 
> > On 01/18/17 00:34, O. Hartmann wrote:  
> >> On Thu, 5 Jan 2017 20:17:56 -0700
> >> Sean Bruno  wrote:  
> >>>  
> >> On a Fujitsu Celsius M740, the "em0" device gets stuck on heavy I/O. I can
> >> still trigger this behaviour on recent CURRENT (12.0-CURRENT #17 r312369:
> >> Wed Jan 18 06:18:45 CET 2017 amd64) by rsync'ing a large poudriere ports
> >> repository onto a remote NFSv4 fileserver. The freeze always occur on large
> >> tarballs.
> >>
> >> Again, here is the pciconf output of the device: 
> >>
> >> em0@pci0:0:25:0:class=0x02 card=0x11ed1734 chip=0x153a8086
> >> rev=0x05 hdr=0x00 vendor = 'Intel Corporation'
> >> device = 'Ethernet Connection I217-LM'
> >> class  = network
> >> subclass   = ethernet
> >> bar   [10] = type Memory, range 32, base 0xfb30, size 131072,
> >> enabled bar   [14] = type Memory, range 32, base 0xfb339000, size 4096,
> >> enabled bar   [18] = type I/O Port, range 32, base 0xf020, size 32, enabled
> >>
> >> On another box. equipted with a dual-port Intel i350 NIC, the igb0 and
> >> igb1 do have negotiation problems with several types of switches (in my
> >> SoHo environment, I use a Netgear GS110TP, at work there are several types
> >> of Cisco Catalyst 3XXX types). The igbX very often fall back to 100MBit/s.
> >>
> >> Since yesterday, the igbX on that specific i350 basesd NIC (we have plentz
> >> of them and they show similar phenomena with FreeBSD), although the switch
> >> reports an uplink with 1 GBit, FreeBSD CURRENT shows this weird crap
> >> message: 
> >>> igb0: flags=8843 metric 0 mtu
> >>> 1500
> >>> options=653dbb
> >>> ether xx:xx:xx:xx:xx:xx inet 192.168.0.111 netmask 0xff00 broadcast
> >>> 192.168.0.255 nd6 options=29
> >>>media: Ethernet autoselect (100baseTX )
> >>>status: active  
> >>  
> 
> I just checked my test machines (which are auto/auto on the Juniper
> EX4200 switches in use) and I see them come up with 1000baseTX.  Do you
> set any options in /etc/rc.conf?
> 
> sean
> 

No, I don't.

The line is:
ifconfig_igb0="inet 192.168.0.10 netmask 0xff00"

Nothing else.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/18/17 08:20, O. Hartmann wrote:
> On Wed, 18 Jan 2017 07:59:17 -0700
> Sean Bruno  wrote:
> 
>> On 01/18/17 07:41, Sean Bruno wrote:
>>>
>>>
>>> On 01/18/17 00:34, O. Hartmann wrote:  
 On Thu, 5 Jan 2017 20:17:56 -0700
 Sean Bruno  wrote:  
>  
 On a Fujitsu Celsius M740, the "em0" device gets stuck on heavy I/O. I can
 still trigger this behaviour on recent CURRENT (12.0-CURRENT #17 r312369:
 Wed Jan 18 06:18:45 CET 2017 amd64) by rsync'ing a large poudriere ports
 repository onto a remote NFSv4 fileserver. The freeze always occur on large
 tarballs.

 Again, here is the pciconf output of the device: 

 em0@pci0:0:25:0:class=0x02 card=0x11ed1734 chip=0x153a8086
 rev=0x05 hdr=0x00 vendor = 'Intel Corporation'
 device = 'Ethernet Connection I217-LM'
 class  = network
 subclass   = ethernet
 bar   [10] = type Memory, range 32, base 0xfb30, size 131072,
 enabled bar   [14] = type Memory, range 32, base 0xfb339000, size 4096,
 enabled bar   [18] = type I/O Port, range 32, base 0xf020, size 32, enabled

 On another box. equipted with a dual-port Intel i350 NIC, the igb0 and
 igb1 do have negotiation problems with several types of switches (in my
 SoHo environment, I use a Netgear GS110TP, at work there are several types
 of Cisco Catalyst 3XXX types). The igbX very often fall back to 100MBit/s.

 Since yesterday, the igbX on that specific i350 basesd NIC (we have plentz
 of them and they show similar phenomena with FreeBSD), although the switch
 reports an uplink with 1 GBit, FreeBSD CURRENT shows this weird crap
 message: 
> igb0: flags=8843 metric 0 mtu
> 1500
> options=653dbb
> ether xx:xx:xx:xx:xx:xx inet 192.168.0.111 netmask 0xff00 broadcast
> 192.168.0.255 nd6 options=29
>media: Ethernet autoselect (100baseTX )
>status: active  
  
>>
>> I just checked my test machines (which are auto/auto on the Juniper
>> EX4200 switches in use) and I see them come up with 1000baseTX.  Do you
>> set any options in /etc/rc.conf?
>>
>> sean
>>
> 
> No, I don't.
> 
> The line is:
> ifconfig_igb0="inet 192.168.0.10 netmask 0xff00"
> 
> Nothing else.
> 

Ok, good.  Definitely a regression.

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/18/17 07:41, Sean Bruno wrote:
> 
> 
> On 01/18/17 00:34, O. Hartmann wrote:
>> On Thu, 5 Jan 2017 20:17:56 -0700
>> Sean Bruno  wrote:
>>>
>> On a Fujitsu Celsius M740, the "em0" device gets stuck on heavy I/O. I can
>> still trigger this behaviour on recent CURRENT (12.0-CURRENT #17 r312369: Wed
>> Jan 18 06:18:45 CET 2017 amd64) by rsync'ing a large poudriere ports
>> repository onto a remote NFSv4 fileserver. The freeze always occur on large
>> tarballs.
>>
>> Again, here is the pciconf output of the device: 
>>
>> em0@pci0:0:25:0:class=0x02 card=0x11ed1734 chip=0x153a8086
>> rev=0x05 hdr=0x00 vendor = 'Intel Corporation'
>> device = 'Ethernet Connection I217-LM'
>> class  = network
>> subclass   = ethernet
>> bar   [10] = type Memory, range 32, base 0xfb30, size 131072, enabled
>> bar   [14] = type Memory, range 32, base 0xfb339000, size 4096, enabled
>> bar   [18] = type I/O Port, range 32, base 0xf020, size 32, enabled
>>
>> On another box. equipted with a dual-port Intel i350 NIC, the igb0 and igb1 
>> do
>> have negotiation problems with several types of switches (in my SoHo
>> environment, I use a Netgear GS110TP, at work there are several types of 
>> Cisco
>> Catalyst 3XXX types). The igbX very often fall back to 100MBit/s.
>>
>> Since yesterday, the igbX on that specific i350 basesd NIC (we have plentz of
>> them and they show similar phenomena with FreeBSD), although the switch 
>> reports
>> an uplink with 1 GBit, FreeBSD CURRENT shows this weird crap message:
>>
>>> igb0: flags=8843 metric 0 mtu
>>> 1500
>>> options=653dbb
>>> ether xx:xx:xx:xx:xx:xx inet 192.168.0.111 netmask 0xff00 broadcast
>>> 192.168.0.255 nd6 options=29
>>>media: Ethernet autoselect (100baseTX )
>>>status: active
>>

I just checked my test machines (which are auto/auto on the Juniper
EX4200 switches in use) and I see them come up with 1000baseTX.  Do you
set any options in /etc/rc.conf?

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/18/17 00:34, O. Hartmann wrote:
> On Thu, 5 Jan 2017 20:17:56 -0700
> Sean Bruno  wrote:
> 
>> tl;dr --> igbX devices will become emX devices
>>
>> We're about to commit an update to sys/dev/e1000 that will implement and
>> activate IFLIB for em(4), lem(4) & igb(4) and would appreciate all folks
>> who can test and poke at the drivers to do so this week.  This will have
>> some really great changes for performance and standardization that have
>> been bouncing around inside of various FreeBSD shops that have been
>> collaborating with Matt Macy over the last year.
>>
>> This will implement multiple queues for certain em(4) devices that are
>> capable of such things and add some new sysctl's for you to poke at in
>> your monitoring tools.
>>
>> Due to limitations of device registration, igbX devices will become emX
>> devices.  So, you'll need to make a minor update to your rc.conf and
>> scripts that manipulate the network devices.
>>
>> UPDATING will be bumped to reflect these changes.
>>
>> MFC to stable/11 will have a legacy implementation that doesn't use
>> IFLIB for compatibility reasons.
>>
>> A documentation and man page update will follow in the next few days
>> explaining how to work with the changed driver.
>>
>> sean
>>
>> bcc net@ current@ re@
>>
>>
>>
> On a Fujitsu Celsius M740, the "em0" device gets stuck on heavy I/O. I can
> still trigger this behaviour on recent CURRENT (12.0-CURRENT #17 r312369: Wed
> Jan 18 06:18:45 CET 2017 amd64) by rsync'ing a large poudriere ports
> repository onto a remote NFSv4 fileserver. The freeze always occur on large
> tarballs.
> 
> Again, here is the pciconf output of the device: 
> 
> em0@pci0:0:25:0:class=0x02 card=0x11ed1734 chip=0x153a8086
> rev=0x05 hdr=0x00 vendor = 'Intel Corporation'
> device = 'Ethernet Connection I217-LM'
> class  = network
> subclass   = ethernet
> bar   [10] = type Memory, range 32, base 0xfb30, size 131072, enabled
> bar   [14] = type Memory, range 32, base 0xfb339000, size 4096, enabled
> bar   [18] = type I/O Port, range 32, base 0xf020, size 32, enabled
> 
> On another box. equipted with a dual-port Intel i350 NIC, the igb0 and igb1 do
> have negotiation problems with several types of switches (in my SoHo
> environment, I use a Netgear GS110TP, at work there are several types of Cisco
> Catalyst 3XXX types). The igbX very often fall back to 100MBit/s.
> 
> Since yesterday, the igbX on that specific i350 basesd NIC (we have plentz of
> them and they show similar phenomena with FreeBSD), although the switch 
> reports
> an uplink with 1 GBit, FreeBSD CURRENT shows this weird crap message:
> 
>> igb0: flags=8843 metric 0 mtu
>> 1500
>> options=653dbb
>> ether xx:xx:xx:xx:xx:xx inet 192.168.0.111 netmask 0xff00 broadcast
>> 192.168.0.255 nd6 options=29
>>media: Ethernet autoselect (100baseTX )
>>status: active
> 
> I haven't checked whether FreeBSD lies or the switch lies about the linkspeed,
> but will do next time I have access to the box.
> 
> 
> regards,
> Oliver
> 


Ugh.  Ok.  Investigating the link issue, that's gross.

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[O. Hartmann]

On Thu, 5 Jan 2017 20:17:56 -0700
Sean Bruno  wrote:

> tl;dr --> igbX devices will become emX devices
> 
> We're about to commit an update to sys/dev/e1000 that will implement and
> activate IFLIB for em(4), lem(4) & igb(4) and would appreciate all folks
> who can test and poke at the drivers to do so this week.  This will have
> some really great changes for performance and standardization that have
> been bouncing around inside of various FreeBSD shops that have been
> collaborating with Matt Macy over the last year.
> 
> This will implement multiple queues for certain em(4) devices that are
> capable of such things and add some new sysctl's for you to poke at in
> your monitoring tools.
> 
> Due to limitations of device registration, igbX devices will become emX
> devices.  So, you'll need to make a minor update to your rc.conf and
> scripts that manipulate the network devices.
> 
> UPDATING will be bumped to reflect these changes.
> 
> MFC to stable/11 will have a legacy implementation that doesn't use
> IFLIB for compatibility reasons.
> 
> A documentation and man page update will follow in the next few days
> explaining how to work with the changed driver.
> 
> sean
> 
> bcc net@ current@ re@
> 
> 
> 
On a Fujitsu Celsius M740, the "em0" device gets stuck on heavy I/O. I can
still trigger this behaviour on recent CURRENT (12.0-CURRENT #17 r312369: Wed
Jan 18 06:18:45 CET 2017 amd64) by rsync'ing a large poudriere ports
repository onto a remote NFSv4 fileserver. The freeze always occur on large
tarballs.

Again, here is the pciconf output of the device: 

em0@pci0:0:25:0:class=0x02 card=0x11ed1734 chip=0x153a8086
rev=0x05 hdr=0x00 vendor = 'Intel Corporation'
device = 'Ethernet Connection I217-LM'
class  = network
subclass   = ethernet
bar   [10] = type Memory, range 32, base 0xfb30, size 131072, enabled
bar   [14] = type Memory, range 32, base 0xfb339000, size 4096, enabled
bar   [18] = type I/O Port, range 32, base 0xf020, size 32, enabled

On another box. equipted with a dual-port Intel i350 NIC, the igb0 and igb1 do
have negotiation problems with several types of switches (in my SoHo
environment, I use a Netgear GS110TP, at work there are several types of Cisco
Catalyst 3XXX types). The igbX very often fall back to 100MBit/s.

Since yesterday, the igbX on that specific i350 basesd NIC (we have plentz of
them and they show similar phenomena with FreeBSD), although the switch reports
an uplink with 1 GBit, FreeBSD CURRENT shows this weird crap message:

> igb0: flags=8843 metric 0 mtu
> 1500
> options=653dbb
> ether xx:xx:xx:xx:xx:xx inet 192.168.0.111 netmask 0xff00 broadcast
> 192.168.0.255 nd6 options=29
>media: Ethernet autoselect (100baseTX )
>status: active

I haven't checked whether FreeBSD lies or the switch lies about the linkspeed,
but will do next time I have access to the box.


regards,
Oliver
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

 >  A flame graph for the core cycle count and a flame graph with cache miss 
 > stats from pmc would be a great start.
 >  
 > 
 > ​I didn't know the exact event name to use for cache miss stats, but here 
 > are the flame graphs for CPU_CLK_UNHALTED_CORE:
 > http://dev.bsdrp.net/netgate.r311848.CPU_CLK_UNHALTED_CORE.svg
 > http://dev.bsdrp.net/netgate.r311849.CPU_CLK_UNHALTED_CORE.svg

Thanks. Having twice as many txqs would definitely help. It's also clear that 
there may be some sort of peformance issue in iflib_txq_drain. Although it 
could just be non-stop cache misses on the packet headers.

-M

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/11/17 15:44, Sean Bruno wrote:
> 
>> My tunning are (same for both test):
>> hw.igb.rxd="2048" (it should be useless now)
>> hw.igb.txd="2048" (it should be useless now)
>> hw.em.rxd="2048"
>> hw.em.txd="2048"
>> hw.igb.rx_process_limit="-1" (It should be useless now too)
>> hw.em.rx_process_limit="-1"
>>
>> dev.igb.2.fc=0
>> dev.igb.3.fc=0
>>
>> I can generate profiling data for you: what kind of data do you want ?
> 
> 
> Specifically, you may want to adjust these:
> 
> dev.em.0.iflib.override_nrxds: 0
> dev.em.0.iflib.override_ntxds: 0
> 
> dev.em.0.iflib.override_nrxqs: 0
> dev.em.0.iflib.override_ntxqs: 0
> 
> sean
> 

dev.igb.0  but you get the point.

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

> My tunning are (same for both test):
> hw.igb.rxd="2048" (it should be useless now)
> hw.igb.txd="2048" (it should be useless now)
> hw.em.rxd="2048"
> hw.em.txd="2048"
> hw.igb.rx_process_limit="-1" (It should be useless now too)
> hw.em.rx_process_limit="-1"
> 
> dev.igb.2.fc=0
> dev.igb.3.fc=0
> 
> I can generate profiling data for you: what kind of data do you want ?


Specifically, you may want to adjust these:

dev.em.0.iflib.override_nrxds: 0
dev.em.0.iflib.override_ntxds: 0

dev.em.0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntxqs: 0

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Wed, Jan 11, 2017 at 9:40 PM, Matthew Macy  wrote:

>
>  >  > I can generate profiling data for you: what kind of data do you want
> ?
>  >  >
>  >
>
> A flame graph for the core cycle count and a flame graph with cache miss
> stats from pmc would be a great start.
>
>
​I didn't know the exact event name to use for cache miss stats, but here
are the flame graphs for CPU_CLK_UNHALTED_CORE:
http://dev.bsdrp.net/netgate.r311848.CPU_CLK_UNHALTED_CORE.svg
http://dev.bsdrp.net/netgate.r311849.CPU_CLK_UNHALTED_CORE.svg
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

 > >  > My tunning are (same for both test):
 > >  > hw.igb.rxd="2048" (it should be useless now)
 > >  > hw.igb.txd="2048" (it should be useless now)
 > 
 > Matt: I think he meant "useless now" because there is no igb, and the
 > below hw.em version covers it.


No. igb still exists and the old tunables no work.
-M

 > >  > hw.em.rxd="2048"
 > >  > hw.em.txd="2048"
 > >  > hw.igb.rx_process_limit="-1" (It should be useless now too)
 > >  > hw.em.rx_process_limit="-1"
 > >  > 
 > >  > dev.igb.2.fc=0
 > >  > dev.igb.3.fc=0
 > >  > 
 > >  > I can generate profiling data for you: what kind of data do you want ?
 > >  >  
 > > 
 > > 
 > > ___
 > > freebsd-current@freebsd.org mailing list
 > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
 > > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
 > > 
 > 
 > 
 > -- 
 > Allan Jude
 > ___
 > freebsd-current@freebsd.org mailing list
 > https://lists.freebsd.org/mailman/listinfo/freebsd-current
 > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Allan Jude]

On 2017-01-11 15:37, Matthew Macy wrote:
> You can still explicitly set the number of descriptors. It is now reported 
> under the dev sysctl tree. dev...
> 
> -M
> 
> 
>   On Wed, 11 Jan 2017 12:34:23 -0800 Olivier Cochard-Labbé 
>  wrote  
>  > 
>  > On Wed, Jan 11, 2017 at 9:13 PM, Matthew Macy  wrote:
>  > 
>  >   > Hmmm ... did your old tests do 4 or 8 queues on this hardware?
>  >   >
>  >   > Did the old tests run 1024 tx/rx slots or the max 4096?
>  >  
>  >  That's a great point, only having one thread per core could easily 
> account for this. I'm hoping Sean can make txq != rxq work so that you can 
> have 8txqs and 4 rxqs.
>  >  
>  > 
>  > ​The netgate RCC-VE 4860 is a 4 cores atom C2558E, and I'm using 2 of the 
> 4 Gigabit Intel i350 ports.
>  > Lab detail:
>  > 
> https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_netgate_rcc-ve_4860
>  > 
>  > My tunning are (same for both test):
>  > hw.igb.rxd="2048" (it should be useless now)
>  > hw.igb.txd="2048" (it should be useless now)

Matt: I think he meant "useless now" because there is no igb, and the
below hw.em version covers it.

>  > hw.em.rxd="2048"
>  > hw.em.txd="2048"
>  > hw.igb.rx_process_limit="-1" (It should be useless now too)
>  > hw.em.rx_process_limit="-1"
>  > 
>  > dev.igb.2.fc=0
>  > dev.igb.3.fc=0
>  > 
>  > I can generate profiling data for you: what kind of data do you want ?
>  >  
> 
> 
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 


-- 
Allan Jude
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Wed, Jan 11, 2017 at 9:13 PM, Matthew Macy  wrote:

>
>  > Hmmm ... did your old tests do 4 or 8 queues on this hardware?
>  >
>  > Did the old tests run 1024 tx/rx slots or the max 4096?
>
> That's a great point, only having one thread per core could easily account
> for this. I'm hoping Sean can make txq != rxq work so that you can have
> 8txqs and 4 rxqs.
>
>
> ​The netgate RCC-VE 4860 is a 4 cores atom C2558E, and I'm using 2 of the
4 Gigabit Intel i350 ports.
Lab detail:
https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_netgate_rcc-ve_4860

My tunning are (same for both test):
hw.igb.rxd="2048" (it should be useless now)
hw.igb.txd="2048" (it should be useless now)
hw.em.rxd="2048"
hw.em.txd="2048"
hw.igb.rx_process_limit="-1" (It should be useless now too)
hw.em.rx_process_limit="-1"

dev.igb.2.fc=0
dev.igb.3.fc=0

I can generate profiling data for you: what kind of data do you want ?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

 >  > I can generate profiling data for you: what kind of data do you want ?
 >  >  
 > 
 
A flame graph for the core cycle count and a flame graph with cache miss stats 
from pmc would be a great start.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

You can still explicitly set the number of descriptors. It is now reported 
under the dev sysctl tree. dev...

-M


  On Wed, 11 Jan 2017 12:34:23 -0800 Olivier Cochard-Labbé 
 wrote  
 > 
 > On Wed, Jan 11, 2017 at 9:13 PM, Matthew Macy  wrote:
 > 
 >   > Hmmm ... did your old tests do 4 or 8 queues on this hardware?
 >   >
 >   > Did the old tests run 1024 tx/rx slots or the max 4096?
 >  
 >  That's a great point, only having one thread per core could easily account 
 > for this. I'm hoping Sean can make txq != rxq work so that you can have 
 > 8txqs and 4 rxqs.
 >  
 > 
 > ​The netgate RCC-VE 4860 is a 4 cores atom C2558E, and I'm using 2 of the 4 
 > Gigabit Intel i350 ports.
 > Lab detail:
 > https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_netgate_rcc-ve_4860
 > 
 > My tunning are (same for both test):
 > hw.igb.rxd="2048" (it should be useless now)
 > hw.igb.txd="2048" (it should be useless now)
 > hw.em.rxd="2048"
 > hw.em.txd="2048"
 > hw.igb.rx_process_limit="-1" (It should be useless now too)
 > hw.em.rx_process_limit="-1"
 > 
 > dev.igb.2.fc=0
 > dev.igb.3.fc=0
 > 
 > I can generate profiling data for you: what kind of data do you want ?
 >  


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

  On Wed, 11 Jan 2017 12:02:06 -0800 Sean Bruno  wrote 
 
 >  
 >  
 > On 01/11/17 12:47, Olivier Cochard-Labbé wrote: 
 > > On Wed, Jan 11, 2017 at 4:17 PM, Sean Bruno  > > wrote: 
 > >  
 > >  
 > >  
 > > Olivier: 
 > >  
 > > Give this a quick try.  This isn't the correct way to do this, but I 
 > > want to see if I'm on the right path: 
 > >  
 > >  
 > > ​thanks, it fix the problem, I've got back the 4 queues:​ 
 > >  
 > > ​igb2:  port 0x3000-0x301f 
 > > mem 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at device 20.0 on 
 > > pci0 
 > > igb2: attach_pre capping queues at 8 
 > > igb2: using 1024 tx descriptors and 1024 rx descriptors 
 > > igb2: msix_init qsets capped at 8 
 > > igb2: pxm cpus: 4 queue msgs: 9 admincnt: 1 
 > > igb2: using 4 rx queues 4 tx queues 
 > > igb2: Using MSIX interrupts with 5 vectors 
 > > igb2: allocated for 4 tx_queues 
 > > igb2: allocated for 4 rx_queues 
 > > igb2: Ethernet address: 00:08:a2:09:33:da 
 > > igb2: netmap queues/slots: TX 4/1024, RX 4/1024 
 > > ​ 
 > > In forwarding mode, I measure about 10% performance drop with this new 
 > > drivers on this hardware: 
 > >  
 > > x head r311848: packets per second 
 > > + head r311849 and BAR patch: packets per second 
 > > +--+
 > >  
 > > |++++ +   xxx x   
 > > x| 
 > > ||__M__A|  
 > > | 
 > > | |___AM__|
 > > | 
 > > +--+
 > >  
 > > N   Min   MaxMedian   Avg
 > > Stddev 
 > > x   5924170943071927509  931612.1 
 > > 8096.8269 
 > > +   5831452  845929.5840940  838730.5 
 > > 6413.5602 
 > > Difference at 95.0% confidence 
 > > -92881.6 +/- 10652.2 
 > > -9.96999% +/- 1.07481% 
 > > (Student's t, pooled s = 7303.85) 
 > >  
 > > Regards, 
 > >  
 > > Olivier 
 > >  
 >  
 >  
 > Hmmm ... did your old tests do 4 or 8 queues on this hardware? 
 >  
 > Did the old tests run 1024 tx/rx slots or the max 4096? 

That's a great point, only having one thread per core could easily account for 
this. I'm hoping Sean can make txq != rxq work so that you can have 8txqs and 4 
rxqs.

-M

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/11/17 12:47, Olivier Cochard-Labbé wrote:
> On Wed, Jan 11, 2017 at 4:17 PM, Sean Bruno  > wrote:
> 
> 
> 
> Olivier:
> 
> Give this a quick try.  This isn't the correct way to do this, but I
> want to see if I'm on the right path:
> 
> 
> ​thanks, it fix the problem, I've got back the 4 queues:​
> 
> ​igb2:  port 0x3000-0x301f
> mem 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at device 20.0 on
> pci0
> igb2: attach_pre capping queues at 8
> igb2: using 1024 tx descriptors and 1024 rx descriptors
> igb2: msix_init qsets capped at 8
> igb2: pxm cpus: 4 queue msgs: 9 admincnt: 1
> igb2: using 4 rx queues 4 tx queues
> igb2: Using MSIX interrupts with 5 vectors
> igb2: allocated for 4 tx_queues
> igb2: allocated for 4 rx_queues
> igb2: Ethernet address: 00:08:a2:09:33:da
> igb2: netmap queues/slots: TX 4/1024, RX 4/1024
> ​
> In forwarding mode, I measure about 10% performance drop with this new
> drivers on this hardware:
> 
> x head r311848: packets per second
> + head r311849 and BAR patch: packets per second
> +--+
> |++++ +   xxx x   x|
> ||__M__A|  |
> | |___AM__||
> +--+
> N   Min   MaxMedian   AvgStddev
> x   5924170943071927509  931612.1 8096.8269
> +   5831452  845929.5840940  838730.5 6413.5602
> Difference at 95.0% confidence
> -92881.6 +/- 10652.2
> -9.96999% +/- 1.07481%
> (Student's t, pooled s = 7303.85)
> 
> Regards,
> 
> Olivier
> 


Hmmm ... did your old tests do 4 or 8 queues on this hardware?

Did the old tests run 1024 tx/rx slots or the max 4096?

sean



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Wed, Jan 11, 2017 at 4:17 PM, Sean Bruno  wrote:

>
>
> Olivier:
>
> Give this a quick try.  This isn't the correct way to do this, but I
> want to see if I'm on the right path:
>

​thanks, it fix the problem, I've got back the 4 queues:​

​igb2:  port 0x3000-0x301f
mem 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at device 20.0 on
pci0
igb2: attach_pre capping queues at 8
igb2: using 1024 tx descriptors and 1024 rx descriptors
igb2: msix_init qsets capped at 8
igb2: pxm cpus: 4 queue msgs: 9 admincnt: 1
igb2: using 4 rx queues 4 tx queues
igb2: Using MSIX interrupts with 5 vectors
igb2: allocated for 4 tx_queues
igb2: allocated for 4 rx_queues
igb2: Ethernet address: 00:08:a2:09:33:da
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
​
In forwarding mode, I measure about 10% performance drop with this new
drivers on this hardware:

x head r311848: packets per second
+ head r311849 and BAR patch: packets per second
+--+
|++++ +   xxx x   x|
||__M__A|  |
| |___AM__||
+--+
N   Min   MaxMedian   AvgStddev
x   5924170943071927509  931612.1 8096.8269
+   5831452  845929.5840940  838730.5 6413.5602
Difference at 95.0% confidence
-92881.6 +/- 10652.2
-9.96999% +/- 1.07481%
(Student's t, pooled s = 7303.85)

Regards,

Olivier
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

 > 
 > x head r311848: packets per second
 > + head r311849 and BAR patch: packets per second
 > +--+
 > |++++ +   xxx x   x|
 > ||__M__A|  |
 > | |___AM__||
 > +--+
 > N   Min   MaxMedian   AvgStddev
 > x   5924170943071927509  931612.1 8096.8269
 > +   5831452  845929.5840940  838730.5 6413.5602
 > Difference at 95.0% confidence
 > -92881.6 +/- 10652.2
 > -9.96999% +/- 1.07481%
 > (Student's t, pooled s = 7303.85)

I apologize. I've only had profiling data for NFLX's read-mostly workload on 
which it (or at least ixl) is 10% faster.

Could you please detail your setup for us so that we can reproduce and get 
profiling data?

Thanks in advance.
-M

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/11/17 05:54, Matthew Macy wrote:
> 
> 
> 
>   On Wed, 11 Jan 2017 01:23:46 -0800 Olivier Cochard-Labbé 
>  wrote  
>  > On Tue, Jan 10, 2017 at 4:31 AM, Sean Bruno  wrote:
>  > 
>  > >
>  > > I've updated sys/dev/e1000 at svn R311849 to match Matt Macy's work on
>  > > IFLIB in the kernel.
>  > >
>  > > At this point, the driver deviates from Intel's code dramatically and
>  > > you now get to yell directly into the freebsd-net@ megaphone for things
>  > > that I may have broken.
>  > >
>  > >
>  > >
>  > ​I've got problem with this new drivers regarding number of queues used on
>  > a Netgate RCC-VE 4860 (Intel i354 NIC).
>  > ​Only one queue in place of 4 (on a 4 cores proc) previously: Performance
>  > drops dramatically.
>  > 

>  > igb2:  port 0x3000-0x301f mem
>  > 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
>  > igb2: attach_pre capping queues at 8
>  > igb2: using 1024 tx descriptors and 1024 rx descriptors
>  > igb2: msix_init qsets capped at 8
>  > igb2: Unable to map MSIX table
> 
> It has the wrong msix bar for your device. I'll look in to it.
> 


Olivier:

Give this a quick try.  This isn't the correct way to do this, but I
want to see if I'm on the right path:
Index: sys/net/iflib.c
===
--- sys/net/iflib.c (revision 311875)
+++ sys/net/iflib.c (working copy)
@@ -4721,7 +4721,7 @@
if_softc_ctx_t scctx = >ifc_softc_ctx;
int vectors, queues, rx_queues, tx_queues, queuemsgs, msgs;
int iflib_num_tx_queues, iflib_num_rx_queues;
-   int err, admincnt, bar;
+   int err, admincnt, bar, use_different_bar;

iflib_num_tx_queues = scctx->isc_ntxqsets;
iflib_num_rx_queues = scctx->isc_nrxqsets;
@@ -4729,6 +4729,16 @@
device_printf(dev, "msix_init qsets capped at %d\n", 
iflib_num_tx_queues);

bar = ctx->ifc_softc_ctx.isc_msix_bar;
+
+/*
+** Some new devices, as with ixgbe, now may
+** use a different BAR, so we need to keep
+** track of which is used.
+*/
+   use_different_bar = pci_read_config(dev, bar, 4);
+   if (use_different_bar == 0)
+   bar += 4;
+
admincnt = sctx->isc_admin_intrcnt;
/* Override by tuneable */
if (enable_msix == 0)





signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Matthew Macy]

  On Wed, 11 Jan 2017 01:23:46 -0800 Olivier Cochard-Labbé 
 wrote  
 > On Tue, Jan 10, 2017 at 4:31 AM, Sean Bruno  wrote:
 > 
 > >
 > > I've updated sys/dev/e1000 at svn R311849 to match Matt Macy's work on
 > > IFLIB in the kernel.
 > >
 > > At this point, the driver deviates from Intel's code dramatically and
 > > you now get to yell directly into the freebsd-net@ megaphone for things
 > > that I may have broken.
 > >
 > >
 > >
 > ​I've got problem with this new drivers regarding number of queues used on
 > a Netgate RCC-VE 4860 (Intel i354 NIC).
 > ​Only one queue in place of 4 (on a 4 cores proc) previously: Performance
 > drops dramatically.
 > 
 > BEFORE (r311848) :
 > 
 > igb2:  port
 > 0x3000-0x301f mem 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at
 > device 20.0 on pci0
 > igb2: Using MSIX interrupts with 5 vectors
 > igb2: Ethernet address: 00:08:a2:09:33:da
 > igb2: Bound queue 0 to cpu 0
 > igb2: Bound queue 1 to cpu 1
 > igb2: Bound queue 2 to cpu 2
 > igb2: Bound queue 3 to cpu 3
 > igb2: netmap queues/slots: TX 4/2048, RX 4/2048
 > 
 > AFTER (r3118489):
 > 
 > igb2:  port 0x3000-0x301f mem
 > 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
 > igb2: attach_pre capping queues at 8
 > igb2: using 1024 tx descriptors and 1024 rx descriptors
 > igb2: msix_init qsets capped at 8
 > igb2: Unable to map MSIX table

It has the wrong msix bar for your device. I'll look in to it.


-M

 > igb2: Using an MSI interrupt
 > igb2: allocated for 1 tx_queues
 > igb2: allocated for 1 rx_queues
 > igb2: Ethernet address: 00:08:a2:09:33:da
 > igb2: netmap queues/slots: TX 1/1024, RX 1/1024
 > 
 > ​NIC information:​
 > 
 > ​igb2@pci0:0:20:0:   class=0x02 card=0x1f418086 chip=0x1f418086
 > rev=0x03 hdr=0x00
 > vendor = 'Intel Corporation'
 > device = 'Ethernet Connection I354'
 > class  = network
 > subclass   = ethernet
 > bar   [10] = type Memory, range 64, base 0xdfea, size 131072,
 > enabled
 > bar   [18] = type I/O Port, range 32, base 0x3000, size 32, enabled
 > bar   [20] = type Memory, range 64, base 0xdff24000, size 16384, enabled
 > cap 01[40] = powerspec 3  supports D0 D3  current D0
 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks
 > cap 11[70] = MSI-X supports 10 messages, enabled
 >  Table in map 0x20[0x0], PBA in map 0x20[0x2000]
 > cap 10[a0] = PCI-Express 2 root endpoint max data 128(512) FLR RO NS
 > ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
 > ecap 0003[140] = Serial 1 0008a20933da
 > ecap 0017[1a0] = TPH Requester 1
 > ecap 000d[1d0] = ACS 1
 > igb3@pci0:0:20:1:   class=0x02 card=0x1f4
 > ___
 > freebsd-current@freebsd.org mailing list
 > https://lists.freebsd.org/mailman/listinfo/freebsd-current
 > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Olivier Cochard-Labbé]

On Tue, Jan 10, 2017 at 4:31 AM, Sean Bruno  wrote:

>
> I've updated sys/dev/e1000 at svn R311849 to match Matt Macy's work on
> IFLIB in the kernel.
>
> At this point, the driver deviates from Intel's code dramatically and
> you now get to yell directly into the freebsd-net@ megaphone for things
> that I may have broken.
>
>
>
​I've got problem with this new drivers regarding number of queues used on
a Netgate RCC-VE 4860 (Intel i354 NIC).
​Only one queue in place of 4 (on a 4 cores proc) previously: Performance
drops dramatically.

BEFORE (r311848) :

igb2:  port
0x3000-0x301f mem 0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at
device 20.0 on pci0
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 00:08:a2:09:33:da
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb2: netmap queues/slots: TX 4/2048, RX 4/2048

AFTER (r3118489):

igb2:  port 0x3000-0x301f mem
0xdfea-0xdfeb,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
igb2: attach_pre capping queues at 8
igb2: using 1024 tx descriptors and 1024 rx descriptors
igb2: msix_init qsets capped at 8
igb2: Unable to map MSIX table
igb2: Using an MSI interrupt
igb2: allocated for 1 tx_queues
igb2: allocated for 1 rx_queues
igb2: Ethernet address: 00:08:a2:09:33:da
igb2: netmap queues/slots: TX 1/1024, RX 1/1024

​NIC information:​

​igb2@pci0:0:20:0:   class=0x02 card=0x1f418086 chip=0x1f418086
rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Connection I354'
class  = network
subclass   = ethernet
bar   [10] = type Memory, range 64, base 0xdfea, size 131072,
enabled
bar   [18] = type I/O Port, range 32, base 0x3000, size 32, enabled
bar   [20] = type Memory, range 64, base 0xdff24000, size 16384, enabled
cap 01[40] = powerspec 3  supports D0 D3  current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 root endpoint max data 128(512) FLR RO NS
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
ecap 0003[140] = Serial 1 0008a20933da
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1d0] = ACS 1
igb3@pci0:0:20:1:   class=0x02 card=0x1f4
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

tl;dir --> you get to keep your igbX devices(thanks jhb), no POLA
violations this week.

I've updated sys/dev/e1000 at svn R311849 to match Matt Macy's work on
IFLIB in the kernel.

At this point, the driver deviates from Intel's code dramatically and
you now get to yell directly into the freebsd-net@ megaphone for things
that I may have broken.

man page updates are coming up next.  Please let us know if this
revision has made things better, worse or none-of-the above on whatever
Intel Gigabit NIC you happen to have lying around.

sean

On 01/05/17 20:17, Sean Bruno wrote:
> tl;dr --> igbX devices will become emX devices
> 
> We're about to commit an update to sys/dev/e1000 that will implement and
> activate IFLIB for em(4), lem(4) & igb(4) and would appreciate all folks
> who can test and poke at the drivers to do so this week.  This will have
> some really great changes for performance and standardization that have
> been bouncing around inside of various FreeBSD shops that have been
> collaborating with Matt Macy over the last year.
> 
> This will implement multiple queues for certain em(4) devices that are
> capable of such things and add some new sysctl's for you to poke at in
> your monitoring tools.
> 
> Due to limitations of device registration, igbX devices will become emX
> devices.  So, you'll need to make a minor update to your rc.conf and
> scripts that manipulate the network devices.
> 
> UPDATING will be bumped to reflect these changes.
> 
> MFC to stable/11 will have a legacy implementation that doesn't use
> IFLIB for compatibility reasons.
> 
> A documentation and man page update will follow in the next few days
> explaining how to work with the changed driver.
> 
> sean
> 
> bcc net@ current@ re@
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[John Baldwin]

On Thursday, January 05, 2017 08:17:56 PM Sean Bruno wrote:
> tl;dr --> igbX devices will become emX devices
> 
> We're about to commit an update to sys/dev/e1000 that will implement and
> activate IFLIB for em(4), lem(4) & igb(4) and would appreciate all folks
> who can test and poke at the drivers to do so this week.  This will have
> some really great changes for performance and standardization that have
> been bouncing around inside of various FreeBSD shops that have been
> collaborating with Matt Macy over the last year.
> 
> This will implement multiple queues for certain em(4) devices that are
> capable of such things and add some new sysctl's for you to poke at in
> your monitoring tools.
> 
> Due to limitations of device registration, igbX devices will become emX
> devices.  So, you'll need to make a minor update to your rc.conf and
> scripts that manipulate the network devices.
> 
> UPDATING will be bumped to reflect these changes.
> 
> MFC to stable/11 will have a legacy implementation that doesn't use
> IFLIB for compatibility reasons.
> 
> A documentation and man page update will follow in the next few days
> explaining how to work with the changed driver.

This is a very invasive change, and seems unnecessary.  The only thing you
can't share between two drivers with different names is the probe routine
(which you would want to be different since they would cover different
PCI ID lists).  That is, you only need:

static int
igb_probe(device_t dev)
{
/* check igb PCI ID list */
}

static int
em_probe(device_t dev)
{
/* check em PCI ID list */
}

static int
foo_attach(device_t dev)
{
   ...
}

static int
foo_detach(device_t dev)
{
   ...
}

static device_method_t igb_methods[] = {
DEVMETHOD(device_probe,   igb_probe),
DEVMETHOD(device_attach,  foo_attach),
/* Other methods all use foo_* */
};

static device_method_t em_methods[] = {
DEVMETHOD(device_probe,   em_probe),
DEVMETHOD(device_attach,  foo_attach),
/* Other methods all use foo_* */
};

static driver_t igb_driver = {
"igb", igb_methods, sizeof(struct foo_softc)
};

static driver_t em_driver = {
"em", em_methods, sizeof(struct foo_softc)
};

DRIVER_MODULE(igb, pci, igb_driver, ...);
DRIVER_MODULE(em, pci, em_driver, ...);

This isn't a huge amount of code to carry around, and seems a very small
price to pay compared to the cost of breaking existing machines (and
existing documentation, so now things have to document <= 11 and >= 12
differently, etc.).

(FWIW, this approach is what cxgbe uses to have the same driver code manage
cxgbe/cxl/cc.)

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Sean Bruno]

On 01/06/17 03:48, Steven Hartland wrote:
> Hmm I'm not sure about everyone else but I we treat emX as legacy
> devices (not used one in years) but igbX is very common here.
> 
> The impact of changing a nic device name is quite a bit more involved
> than just rc.conf it effects other areas too, jails etc so given we can
> loose access to the machine on reboot if everything isn't done right, it
> would be worth considering:
> 
>  * Change emX -> igbX to lower the impact.
>  * Add shims / alias so that operations on the device name going away
>still work.
> 
> What do people think?
> 

We have a "legacy" code implementation that does exactly what you're
describing and we intend on putting that version into 11-stable so that
existing users won't bang their heads against it.

The amount of code in the tree dropped *significantly* when we dropped
this implementation, hence why we wanted to make 12-current a clean break.

sean

bcc matt
> 
> On 06/01/2017 03:17, Sean Bruno wrote:
>> tl;dr --> igbX devices will become emX devices
>>
>> We're about to commit an update to sys/dev/e1000 that will implement and
>> activate IFLIB for em(4), lem(4) & igb(4) and would appreciate all folks
>> who can test and poke at the drivers to do so this week.  This will have
>> some really great changes for performance and standardization that have
>> been bouncing around inside of various FreeBSD shops that have been
>> collaborating with Matt Macy over the last year.
>>
>> This will implement multiple queues for certain em(4) devices that are
>> capable of such things and add some new sysctl's for you to poke at in
>> your monitoring tools.
>>
>> Due to limitations of device registration, igbX devices will become emX
>> devices.  So, you'll need to make a minor update to your rc.conf and
>> scripts that manipulate the network devices.
>>
>> UPDATING will be bumped to reflect these changes.
>>
>> MFC to stable/11 will have a legacy implementation that doesn't use
>> IFLIB for compatibility reasons.
>>
>> A documentation and man page update will follow in the next few days
>> explaining how to work with the changed driver.
>>
>> sean
>>
>> bcc net@ current@ re@
>>
>>
>>
> 
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 



signature.asc
Description: OpenPGP digital signature

Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

[Steven Hartland]

Hmm I'm not sure about everyone else but I we treat emX as legacy 
devices (not used one in years) but igbX is very common here.


The impact of changing a nic device name is quite a bit more involved 
than just rc.conf it effects other areas too, jails etc so given we can 
loose access to the machine on reboot if everything isn't done right, it 
would be worth considering:


 * Change emX -> igbX to lower the impact.
 * Add shims / alias so that operations on the device name going away
   still work.

What do people think?


On 06/01/2017 03:17, Sean Bruno wrote:

tl;dr --> igbX devices will become emX devices

We're about to commit an update to sys/dev/e1000 that will implement and
activate IFLIB for em(4), lem(4) & igb(4) and would appreciate all folks
who can test and poke at the drivers to do so this week.  This will have
some really great changes for performance and standardization that have
been bouncing around inside of various FreeBSD shops that have been
collaborating with Matt Macy over the last year.

This will implement multiple queues for certain em(4) devices that are
capable of such things and add some new sysctl's for you to poke at in
your monitoring tools.

Due to limitations of device registration, igbX devices will become emX
devices.  So, you'll need to make a minor update to your rc.conf and
scripts that manipulate the network devices.

UPDATING will be bumped to reflect these changes.

MFC to stable/11 will have a legacy implementation that doesn't use
IFLIB for compatibility reasons.

A documentation and man page update will follow in the next few days
explaining how to work with the changed driver.

sean

bcc net@ current@ re@





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"