Re: Looking for some old laptops

[Kevin Bowling]

I've picked up a cheap W500 and W510 to fulfil 82567LM and 82577LM.  I
have an 82573L based add in card so I can nix the x60 series.  I think
the 82540EP and 82541GI are close enough to the add in card variants I
have as well.

So just looking for something in the x61 series.

On Tue, Aug 15, 2023 at 7:57 PM Kevin Bowling  wrote:
>
> I am looking for one laptop in each of the Intel Ethernet lines below.
>
> They contain different generations of Intel Ethernet controllers, and
> would be used to expand my testing matrix for Intel Ethernet driver
> development.
>
> I am linking ThinkPads because I am familiar with them and can perform
> advanced repairs on them, but there is no specific need for Thinkpads
> if you have something with a similar Ethernet chipset to offer.
>
> https://www.thinkwiki.org/wiki/Intel_Gigabit_Ethernet_PCI-Express_Controller
> Intel 82573L: T60, T60p, X60, X60s
> Intel 82566MM: R61, R61i, T61, T61p, X61, X61s, X61 Tablet, X300
> Intel 82567LM : R400, T400, T400s, T500, W500, W700, W700ds, X200,
> X200s, X200 Tablet, X301
> Intel 82579LM I already have that covered.
>
> https://www.thinkwiki.org/wiki/Intel_Gigabit_Ethernet_Controller
> Intel 82540EP: R50, R50p, R51, T40, T40p, T41, T41p, T42, T42p, X31, X32
> Intel 82541GI: R51,X40
>
> Regards,
> Kevin

Looking for some old laptops

[Kevin Bowling]

I am looking for one laptop in each of the Intel Ethernet lines below.

They contain different generations of Intel Ethernet controllers, and
would be used to expand my testing matrix for Intel Ethernet driver
development.

I am linking ThinkPads because I am familiar with them and can perform
advanced repairs on them, but there is no specific need for Thinkpads
if you have something with a similar Ethernet chipset to offer.

https://www.thinkwiki.org/wiki/Intel_Gigabit_Ethernet_PCI-Express_Controller
Intel 82573L: T60, T60p, X60, X60s
Intel 82566MM: R61, R61i, T61, T61p, X61, X61s, X61 Tablet, X300
Intel 82567LM : R400, T400, T400s, T500, W500, W700, W700ds, X200,
X200s, X200 Tablet, X301
Intel 82579LM I already have that covered.

https://www.thinkwiki.org/wiki/Intel_Gigabit_Ethernet_Controller
Intel 82540EP: R50, R50p, R51, T40, T40p, T41, T41p, T42, T42p, X31, X32
Intel 82541GI: R51,X40

Regards,
Kevin

Re: CFT: lem(4), em(4) e1000 Ethernet TSO testing

[Kevin Bowling]

Committed as 
https://cgit.freebsd.org/src/commit/?id=f1b5488f7bba7f25a57750f87cbcbccbd5b9d16b

On Tue, Jul 25, 2023 at 7:38 PM Kevin Bowling  wrote:
>
> Hi,
>
> I have been working through various bugs and have come to a point
> where TSO is working on systems I have available for testing.
>
> This results in higher throughput on resource constrained systems, and
> less CPU/power usage on unconstrained systems.
>
> As of this mail, you will need to manually apply
> https://reviews.freebsd.org/D41170 on top of main to use TSO6 on
> em(4).
>
> I plan to enable TSO by default for lem(4) and em(4) during the
> FreeBSD 14 release cycle, so I would appreciate testing to address any
> remaining issues.  Below, a list of chipsets that will be exempt due
> to known issues.
>
> lem(4) exclusions:
> * <82544 (although it does seem ok to manually enable for emulations
> in qemu, virtualbox, etc)
> * 82547
>
> em(4) exclusions.. These chips have a stability workaround for high
> throughput with rapid link-flap applied that results in the TSO engine
> not being able to run at line speed.  Thus, TSO would not be enabled
> by default here:
> * Intel(R) I219-LM and I219-V
> * Intel(R) I219-LM and I219-V (2)
> * Intel(R) I219-LM and I219-V (3)
> * Intel(R) I219-LM and I219-V (4)
> * Intel(R) I219-LM and I219-V (5)
>
> Regards,
> Kevin Bowling

Re: tcp and udp traffic over IPv6 does not work from the latest e1000 git change 918c25677d

[Kevin Bowling]

Hi Cheng,

Have you applied https://reviews.freebsd.org/D41170?

Can you also try 'ifconfig emXX -txcsum6' on the DUT?

On Wed, Jul 26, 2023 at 12:37 PM Cheng Cui  wrote:
>
> Hello Kevin,
>
> TCP and UDP traffic over IPv4 are working, but not over IPv6.
> On a pair of FreeBSD 14.0-CURRENT nodes that are using a kernel containing 
> the latest e1000 git change 918c25677d:
>
> root@s1:~ # uname -a
> FreeBSD s1.testsiftr.fbsd-transport.emulab.net 14.0-CURRENT FreeBSD 
> 14.0-CURRENT amd64 1400093 #0 main-918c25677d-dirty: Sat Jul 22 14:43:31 MDT 
> 2023 
> c...@n1.buildbsd14.fbsd-transport.emulab.net:/usr/obj/usr/src/amd64.amd64/sys/TESTBED-GENERIC
>  amd64
> root@s1:~ #
>
> root@s1:~ # ping6 -c 3 fd00::3
> PING6(56=40+8+8 bytes) fd00::2 --> fd00::3
> 16 bytes from fd00::3, icmp_seq=0 hlim=64 time=0.393 ms
> 16 bytes from fd00::3, icmp_seq=1 hlim=64 time=0.171 ms
> 16 bytes from fd00::3, icmp_seq=2 hlim=64 time=0.276 ms
>
> --- fd00::3 ping6 statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss
> round-trip min/avg/max/std-dev = 0.171/0.280/0.393/0.091 ms
>
> root@s1:~ # ifconfig em4
> em4: flags=1008843 metric 0 
> mtu 1500
> options=4e104bb
> ether 00:04:23:b7:40:ac
> inet 10.1.1.2 netmask 0xff00 broadcast 10.1.1.255
> inet6 fd00::2 prefixlen 64
> inet6 fe80::204:23ff:feb7:40ac%em4 prefixlen 64 scopeid 0x5
> media: Ethernet 1000baseT 
> status: active
> nd6 options=21
>
> root@s1:~ # dmesg | grep em4
> em4:  port 0xacc0-0xacff mem 
> 0xdf3e-0xdf3f irq 101 at device 3.0 on pci10
> em4: EEPROM V15.255-15
> em4: Using 1024 TX descriptors and 1024 RX descriptors
> em4: Ethernet address: 00:04:23:b7:40:ac
> em4: netmap queues/slots: TX 1/1024, RX 1/1024
>
> root@r1:~ # ifconfig em4
> em4: flags=1008843 metric 0 
> mtu 1500
> options=4e104bb
> ether 00:04:23:b7:40:1c
> inet 10.1.1.3 netmask 0xff00 broadcast 10.1.1.255
> inet6 fd00::3 prefixlen 64
> inet6 fe80::204:23ff:feb7:401c%em4 prefixlen 64 scopeid 0x5
> media: Ethernet 1000baseT 
> status: active
> nd6 options=21
>
> root@r1:~ # dmesg | grep em4
> em4:  port 0xacc0-0xacff mem 
> 0xdf3e-0xdf3f irq 101 at device 3.0 on pci10
> em4: EEPROM V15.255-15
> em4: Using 1024 TX descriptors and 1024 RX descriptors
> em4: Ethernet address: 00:04:23:b7:40:1c
> em4: netmap queues/slots: TX 1/1024, RX 1/1024
>
>
> TCP connection timed out
> root@s1:~ # iperf -Vc fd00::3 -n 2K
> 
> Client connecting to fd00::3, TCP port 5001
> TCP window size: 32.0 KByte (default)
> 
> tcp connect failed: Operation timed out
> [  1] local :: port 0 connected with fd00::3 port 5001
>
> server side has no response:
> root@r1:~ # iperf -VsB fd00::3
> 
> Server listening on TCP port 5001
> TCP window size: 64.0 KByte (default)
> 
>
> UDP traffic does not work either, and the UDP server has no response:
> root@s1:~ # iperf -Vc fd00::3 --udp -n 2K
> 
> Client connecting to fd00::3, UDP port 5001
> Sending 1450 byte datagrams, IPG target: 11062.62 us (kalman adjust)
> UDP buffer size: 9.00 KByte (default)
> 
> [  1] local fd00::2 port 54362 connected with fd00::3 port 5001
> [ ID] Interval   Transfer Bandwidth
> [  1] 0.00-0.01 sec  3.42 KBytes  2.36 Mbits/sec
> [  1] Sent 4 datagrams
> [  3] WARNING: did not receive ack of last datagram after 1 tries.
>
> root@r1:~ # iperf -VsB fd00::3 --udp
> 
> Server listening on UDP port 5001
> UDP buffer size: 41.1 KByte (default)
> 
>
>
>
> On a new pair of nodes that use a kernel with backed out the latest em git 
> change 918c25677d. Same em chip 82546EB.
> root@s1:~ # uname -a
> FreeBSD s1.testem.fbsd-transport.emulab.net 14.0-CURRENT FreeBSD 14.0-CURRENT 
> amd64 1400093 #0 22dca7acf7-dirty: Wed Jul 26 08:18:23 MDT 2023 
> c...@n1.emtest.fbsd-transport.emulab.net:/usr/obj/usr/src/amd64.amd64/sys/TESTBED-GENERIC
>  amd64
>
> cc@s1:~ % ifconfig em2
> em2: flags=1008843 metric 0 
> mtu 1500
> options=481009b
> ether 00:04:23:b7:12:be
> inet 10.1.1.2 netmask 0xff00 broadcast 10.1.1.255
> inet6 fd00::2 prefixlen 64
> inet6 fe80::204:23ff:feb7:12be%em2 prefixlen 64 scopeid 0x3
> media: Ethernet 1000baseT 
> status: active
> nd6 options=21
>
> cc@r1:~ % ifconfig em2
> em2: flags=1008843 metric 0 
> mtu 1500
> options=481009b
> ether 00:04:23:b7:15:58
> inet 10.1.1.3 netmask 0xff00 broadcast 10.1.1.255
> inet6 fd00::3 prefixlen 64
> inet6 fe80::204:23ff:feb7:1558%em2 prefixlen 64 scopeid 0x3
> media: Ethernet 1000baseT 
> status: active
> nd6 options=21
>
> cc@s1:~ % ping6 -c 3 fd00::3
> PING6(56=40+8+8 bytes) fd00::2 

Re: CFT: lem(4), em(4) e1000 Ethernet TSO testing

[Kevin Bowling]

On Wed, Jul 26, 2023 at 6:43 AM Cheng Cui  wrote:

Hi Cheng,

> I didn't see your post covering 82541 or 82546 chips. Does this new em(4) 
> change support TSO on these chips? If yes, I would be happy to test it on 
> them.

82541 would be excluded, while 82546 would be a candidate to enable
TSO with my patches (ifconfig em2 tso tso6)

>
> root@s1:~ # dmesg | grep 8254
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> em0:  port 0xdcc0-0xdcff mem 
> 0xdfae-0xdfaf irq 64 at device 7.0 on pci6
> em1:  port 0xccc0-0xccff mem 
> 0xdf8e-0xdf8f irq 65 at device 8.0 on pci7
> em2:  port 0xbcc0-0xbcff mem 
> 0xdf5e-0xdf5f irq 106 at device 4.0 on pci9
> em3:  port 0xbc80-0xbcbf mem 
> 0xdf5c-0xdf5d irq 107 at device 4.1 on pci9
> em4:  port 0xacc0-0xacff mem 
> 0xdf3e-0xdf3f irq 101 at device 3.0 on pci10
> em5:  port 0xac80-0xacbf mem 
> 0xdf3c-0xdf3d irq 102 at device 3.1 on pci10
>
> Best Regards,
> Cheng Cui
>
>
> On Tue, Jul 25, 2023 at 10:38 PM Kevin Bowling  
> wrote:
>>
>> Hi,
>>
>> I have been working through various bugs and have come to a point
>> where TSO is working on systems I have available for testing.
>>
>> This results in higher throughput on resource constrained systems, and
>> less CPU/power usage on unconstrained systems.
>>
>> As of this mail, you will need to manually apply
>> https://reviews.freebsd.org/D41170 on top of main to use TSO6 on
>> em(4).
>>
>> I plan to enable TSO by default for lem(4) and em(4) during the
>> FreeBSD 14 release cycle, so I would appreciate testing to address any
>> remaining issues.  Below, a list of chipsets that will be exempt due
>> to known issues.
>>
>> lem(4) exclusions:
>> * <82544 (although it does seem ok to manually enable for emulations
>> in qemu, virtualbox, etc)
>> * 82547
>>
>> em(4) exclusions.. These chips have a stability workaround for high
>> throughput with rapid link-flap applied that results in the TSO engine
>> not being able to run at line speed.  Thus, TSO would not be enabled
>> by default here:
>> * Intel(R) I219-LM and I219-V
>> * Intel(R) I219-LM and I219-V (2)
>> * Intel(R) I219-LM and I219-V (3)
>> * Intel(R) I219-LM and I219-V (4)
>> * Intel(R) I219-LM and I219-V (5)
>>
>> Regards,
>> Kevin Bowling
>>

Re: CFT: lem(4), em(4) e1000 Ethernet TSO testing

[Kevin Bowling]

On Tue, Jul 25, 2023 at 7:38 PM Kevin Bowling 
wrote:

> Hi,
>
> I have been working through various bugs and have come to a point
> where TSO is working on systems I have available for testing.
>
> This results in higher throughput on resource constrained systems, and
> less CPU/power usage on unconstrained systems.
>
> As of this mail, you will need to manually apply
> https://reviews.freebsd.org/D41170 on top of main to use TSO6 on
> em(4).


I forgot to include instructions:

For testing you can enable with:
ifconfig em0 tso tso6

And disable with:
ifconfig em0 -tso -tso6

tso6 (IPv6) will only applicable to em(4) not lem(4)


>
> I plan to enable TSO by default for lem(4) and em(4) during the
> FreeBSD 14 release cycle, so I would appreciate testing to address any
> remaining issues.  Below, a list of chipsets that will be exempt due
> to known issues.
>
> lem(4) exclusions:
> * <82544 (although it does seem ok to manually enable for emulations
> in qemu, virtualbox, etc)
> * 82547
>
> em(4) exclusions.. These chips have a stability workaround for high
> throughput with rapid link-flap applied that results in the TSO engine
> not being able to run at line speed.  Thus, TSO would not be enabled
> by default here:
> * Intel(R) I219-LM and I219-V
> * Intel(R) I219-LM and I219-V (2)
> * Intel(R) I219-LM and I219-V (3)
> * Intel(R) I219-LM and I219-V (4)
> * Intel(R) I219-LM and I219-V (5)
>
> Regards,
> Kevin Bowling
>

CFT: lem(4), em(4) e1000 Ethernet TSO testing

[Kevin Bowling]

Hi,

I have been working through various bugs and have come to a point
where TSO is working on systems I have available for testing.

This results in higher throughput on resource constrained systems, and
less CPU/power usage on unconstrained systems.

As of this mail, you will need to manually apply
https://reviews.freebsd.org/D41170 on top of main to use TSO6 on
em(4).

I plan to enable TSO by default for lem(4) and em(4) during the
FreeBSD 14 release cycle, so I would appreciate testing to address any
remaining issues.  Below, a list of chipsets that will be exempt due
to known issues.

lem(4) exclusions:
* <82544 (although it does seem ok to manually enable for emulations
in qemu, virtualbox, etc)
* 82547

em(4) exclusions.. These chips have a stability workaround for high
throughput with rapid link-flap applied that results in the TSO engine
not being able to run at line speed.  Thus, TSO would not be enabled
by default here:
* Intel(R) I219-LM and I219-V
* Intel(R) I219-LM and I219-V (2)
* Intel(R) I219-LM and I219-V (3)
* Intel(R) I219-LM and I219-V (4)
* Intel(R) I219-LM and I219-V (5)

Regards,
Kevin Bowling

Re: iflib_timer() vs ixl_admin_timer() race

[Kevin Bowling]

On Wed, Jan 12, 2022 at 10:18 AM Alexander Motin  wrote:

> Thanks, Krzystof,
>
> Grepping now for iflib_admin_intr_deferred() through the sources I see
> the same issue in other Intel NIC drivers, plus bnxt(4).  So the main
> controversy I see is this: either admin intr should not stop and restart
> the callouts (and then question is why it does that now), or if it
> should be so heavy-weight, it should not be called so regularly (and
> then question why so many drivers do it).
>
> In few drivers I've found it even more interesting: iflib_timer() calls
> IFDI_TIMER(), which calls ixgbe_if_timer(), which calls
> iflib_admin_intr_deferred(), which in its turn restart the
> iflib_timer().  Ouroboros. ;)
>

>From memory, I believe that call may be related to NICs without dedicated
admin intrs.  Please keep them in mind when you clean this up.


> On 12.01.2022 11:46, Galazka, Krzysztof wrote:
> > Hi Alexander,
> >
> > Thank you for pointing this out. ixl_admin_timer is used by a callout so
> > I don’t think we can acquire any locks there. IIRC it was added to let
> > tools for NVM update interact with FW while interface is down, so maybe
> > stopping it when interface goes UP would be an option. Let me think this
> > through.
> >
> > Thanks,
> >
> > Krzysiek
> >
> > *From:* Eric Joyner 
> > *Sent:* Wednesday, January 12, 2022 8:22 AM
> > *To:* Alexander Motin 
> > *Cc:* Joyner, Eric ; Galazka, Krzysztof
> > 
> > *Subject:* Re: iflib_timer() vs ixl_admin_timer() race
> >
> > Well, I think the situation is more-or-less as you say -- it's just that
> > the intent of ixl_admin_timer() is supposed to have the adapter's admin
> > queue processed constantly, regardless of interrupt status or down/up
> > status. I think as a shorthand we just called _task_fn_admin because it
> > handles a bunch of other things as well as getting down to calling
> > ixl_if_update_admin_status() which does the admin queue processing. But
> > as you found, I don't think it was originally written to be called
> > periodically on a regular basis like iflib_timer(), so the callouts are
> > interacting badly.
> >
> > My first thought would be to replace the call to
> > iflib_admin_intr_deferred() in ixl_admin_timer() with
> > ixl_if_update_admin_status() while taking the CTX_LOCK(). I'm not sure
> > everything else in _task_fn_admin() needs to run periodically like that
> > in the driver. That would avoid the callouts getting stopped every 500
> > ticks.
> >
> > I CC'd my coworker Krzysztof who currently owns the driver; he might
> > have better thoughts on this.
> >
> > - Eric
> >
> > On Tue, Jan 11, 2022 at 10:46 AM Alexander Motin  > > wrote:
> >
> > Yes.  I've reverted my patch for now to not break anything, but all
> > this
> > situation does not look right for me too, so I'd appreciate your
> look.
> >
> > On 11.01.2022 12:21, Eric Joyner wrote:
> >  > Hi,
> >  >
> >  > I came back from vacation yesterday, but I'll look into this for
> you
> >  > today. It's not a good situation when changing the period of the
> > iflib
> >  > timer breaks something in the driver...
> >  >
> >  > - Eric
> >  >
> >  > On Sun, Jan 9, 2022 at 8:19 PM Alexander Motin  > 
> >  > >> wrote:
> >  >
> >  > Hi Eric,
> >  >
> >  > In 90bc1cf65778 I've done what looked like a trivial
> > optimization.  But
> >  > just after committing it I've noticed weird race it created
> > between
> >  > iflib_timer() and ixl_admin_timer() timers.  I see ixl(4)
> calls
> >  > iflib_admin_intr_deferred() every 0.5s, which calls
> > _task_fn_admin(),
> >  > which every time stops and restart txq->ift_timer callout (AKA
> >  > iflib_timer()), which actually has exactly the same period of
> > 0.5s.  It
> >  > seems before my change iflib_timer() managed to run once for
> > all TX
> >  > queues before being stopped, but after the change due to
> > additional
> >  > jitter many of callouts are getting stopped before firing.
> >  >
> >  > Could you please sched some light for me on what is going on
> > there?
> >  > That race between two callouts looks like potential bug to
> > me, working
> >  > by some coinncedence, especially considering iflib_timer()
> > period is
> >  > tunable.
> >  >
> >  > Thanks in advance.
> >  >
> >  > --
> >  > Alexander Motin
> >  >
> >
> > --
> > Alexander Motin
> >
> > 
> > *Intel Technology Poland sp. z o.o.
> > * ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII
> > Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP
> > 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
> >
> > Ta 

Re: Bug in MAC filter on IGB/if_vlan?

[Kevin Bowling]

On Sat, Nov 6, 2021 at 5:03 PM Rozhuk Ivan  wrote:

> Hi!
>
> I have in rc.conf:
>
> =
> vlans_igb0="vlan77 vlan86 vlan87"
> create_args_vlan87="vlan 87"
> create_args_vlan86="vlan 86"
> create_args_vlan77="vlan 77"
> ifconfig_vlan87="inet 185.44.68.92 netmask 255.255.252.0 mtu 1500 down up"
> ifconfig_vlan87_alias0="link 00:aa:fa:dd:44:55"
> ifconfig_vlan86="DHCP mtu 1500"
> ifconfig_vlan86_alias0="link 00:ff:fa:dd:44:55"
> ifconfig_vlan77="inet 192.168.0.254 netmask 255.255.255.0"
> ifconfig_vlan77_alias0="link 00:0f:43:48:67:fe"
> ifconfig_vlan77_ipv6="inet6 2001:470:2345:555::1/64 prefixlen 64
> auto_linklocal"
> ifconfig_igb0="-lro -tso -vlanhwtso mtu 9000 down up"
>
> =
>
> There is 4 different MAC addresses.
> System is unavailable after boot until: ifconfig igb0 promisc down up
>
> FreeBSD 13 build from fresh sources.
>
>
> Is this a bug or this is normal?


Which chip?  Please post uname -a or git rev.

Why do you have “down up” in the stateful config, please remove “down” and
the MTU change and report back findings

>

Re: Vector Packet Processing (VPP) portability on FreeBSD

[Kevin Bowling]

On Mon, Oct 4, 2021 at 4:35 AM Francois ten Krooden  wrote:
>
> Hi Santiago
>
> The patches we have made is all available on the github fork we made from the 
> VPP repository.
> It is located at https://github.com/ftk-ntq/vpp/tree/freebsd
> So anyone who is interested can find it there.
>
> To make the VFIO support work I unfortunately have no idea.
> I am not exactly sure but I think there is some kernel work required and then 
> an update to DPDK.
> I am not sure how much effort that would be.

I would like to take a look at this, maybe we should discuss it at the
upcoming vendor summit https://wiki.freebsd.org/DevSummit/202111

> Kind Regards
>
> Francois ten Krooden
> Principal Developer
>
> Nanoteq
> Tel: +27 12 672 7000
> Fax: +27 12 665 1343
> Postal: P.O. Box 7991, Centurion, 0046
> Physical: Unit C01, Corporate Park 66, 269 Von Willich Avenue, 
> Centurion
>
> 
> From: Santiago Martinez [s...@codenetworks.net]
> Sent: Wednesday, September 22, 2021 11:22 AM
> To: freebsd-net@freebsd.org
> Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD
>
> Hi Francois, I hope you are doing well.
>
> It is great to hear about work/progress/updates on VPP / DPDK / Netmap
> on FBSD, even if the results are not the best.
>
> Unfortunately, I'm not a developer, so I cannot help much on the matter
> of the missing bits.
> Just wondering if those modifications that your team have done to make
> VPP run can be upstreamed or shared with the community, so maybe we can
> create a VPP package making it easier for others to deploy/test/improve.
>
> On the other hand, do you roughly know how much effort is required to
> make VFIO support at the same level as Linux?
>
> I hope it makes sense.
>
> Best regards.
>
> Santiago
>
>
> On 9/21/21 11:52 AM, Francois ten Krooden wrote:
> > Hi
> >
> > This is just some feedback for those who had an interest in this topic.
> >
> > After spending quite some time on the VPP to FreeBSD porting effort where 
> > we did manage to get VPP working with netmap, and VPP compiling with DPDK; 
> > We realised that there are some big issues that we would need to overcome. 
> > Some of these efforts are not viable for our small team to accomplish in a 
> > reasonable time frame.
> > The main issues that we have found are:
> > - Tests proved that netmap would not deliver the desired performance as it 
> > is currently implemented within VPP. The main issues here are that for 
> > every 256 packets memory seems to be allocated again, also a number of 
> > copies occur in the memory which slows down the performance.
> > - VPP relies on VFIO to map device memory into user space for processing 
> > within the application. This code is implemented in DPDK in the Linux 
> > implementation but in the FreeBSD implementation in DPDK these functions 
> > are stubbed.
> > - To interface with crypto-offloading hardware such as the QAT card from 
> > Intel, or our own card VPP/DPDK also utilize VFIO with the PCI device.
> > - As far as we have been able to see the VFIO support in FreeBSD is not at 
> > the same level as Linux, which would then require additional kernel 
> > development which is not possible in the time frame.
> >
> > Regards
> >
> > Francois ten Krooden
> > Principal Developer
> >
> >
> >  Tel: +27 12 672 7000
> >  Fax: +27 12 665 1343
> >  Postal: P.O. Box 7991, Centurion, 0046
> >  Physical: Unit C01, Corporate Park 66, 269 Von Willich Avenue, 
> > Centurion
> >
> >
> >
> > Important Notice:
> >
> > This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail 
> > legal notice available at:
> > http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
>
>
>
>
>
> Important Notice:
>
> This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail 
> legal notice available at:
> http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
>
>
>

Re: git: 1a72c3d76aea - stable/13 - e1000: always enable PCSD when RSS hashing [Was: TCP6 regression for MTU path on stable/13]

[Kevin Bowling]

On Sat, Sep 25, 2021 at 5:53 PM Harry Schmalzbauer  wrote:
>
> Am 13.09.2021 um 13:18 schrieb Harry Schmalzbauer:
> > Am 13.09.2021 um 12:37 schrieb Andrey V. Elsukov:
> >> 12.09.2021 14:12, Harry Schmalzbauer пишет:
> >>> Will try to further track it down, but in case anybody has an idea,
> >>> what
> >>> change during the last view months in stable/13 could have caused this
> >>> real-world problem regarding resulting TCP6 throughput, I'm happy to
> >>> start testing at that point.
> >>
> >> Hi,
> >>
> >> Take a look at:
> >>
> >>https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255749
> >>https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248005
> >>
> >> does the problem described in these PRs is the same as yours?
> >
> > Hi, thank you very much for your attention!
> > Most likely these are unrelated to the regression I'm suffering from,
> > because these affect 13-release and earlier.
> > Mine arose during the last months.
> > And it seems not to be a jumbo frame problem.
> :
> > Hope to get back to you soon with more info.
>
>
> Since the setup was hard to replicate, it took some time.
> Here's the commit, causing the heavy IPv6 performance drop with Intel
> Powerville and IPv6:
>
> > The branch stable/13 has been updated by kbowling (ports committer):
> >
> > URL:
> > https://cgit.FreeBSD.org/src/commit/?id=1a72c3d76aeafe4422ff20f81c4142efb983b7d7
> >
> > commit 1a72c3d76aeafe4422ff20f81c4142efb983b7d7
> > Author: Kevin Bowling 
> > AuthorDate: 2021-08-16 17:17:34 +
> > Commit: Kevin Bowling 
> > CommitDate: 2021-08-23 16:23:43 +
> >
> > e1000: always enable PCSD when RSS hashing
> >
> > To enable RSS hashing in the NIC, the PCSD bit must be set.
> >
> > By default, this is never set when RXCSUM is disabled - which
> > causes problems higher up in the stack.
> >
> > While here improve the RXCSUM flag assignments when enabling or
> > disabling IFCAP_RXCSUM.
> >
> > See also:
> > https://lists.freebsd.org/pipermail/freebsd-current/2020-May/076148.html
> >
> > Reviewed by:markj, Franco Fichtner ,
> > Stephan de Wit 
> > Obtained from:  OPNsense
> > MFC after:  1 week
> > Differential Revision:  https://reviews.freebsd.org/D31501
> > Co-authored-by: Stephan de Wit 
> > Co-authored-by: Franco Fichtner 
> >
> > (cherry picked from commit 69e8e8ea3d4be9da6b5bc904a444b51958128ff5)
> > :
>
> Noticed and successfully (double-{a8446d412+f72cdea25}) falsified with
> i350 Powerville, device=0x1521.
> *Reverting git: 1a72c3d76aea against today's stable/13(-f72cdea25-dirty)
> sloves the issue, which seems to be IPv6 related only.*
> (kernel  a8446d412 from 21/09/25 shows issue, reverting this commit
> solves it with old kernel too)
>
> Very brief check against IPv4 on identical paths seems to be unaffected,
> but I can't guarantee since v4 isn't in use (where I 1st noticed and
> suffer from) and I just did one comparing in order to narrow down
> (asymmetric FIB setup regarding inet and inet6).
>
> What this made complicated: ng_brige(4), mpd5/pppoe,ppt,bhyve are
> involved as well (and vlan(4), lagg(4) and vtnet(4), etc.), but it seems
> to be just a e1000 driver issue.
> There were many changes/iprovements/cleanups between July and September,
> but I tracked it down as root cause for my IPv6 issue (performance
> dropping from 33MB/s to <=0.3MB/s).
>
>
> That beeing said, it was hard to find the time replicating the setup,
> and I have nothing for a solution.  Haven't semantically checked
> anything yet and didn't do any tests beside my single IPv6 performance
> test.  Contrary to my first suspicion, at least in my clone-lab, it
> isn't MTU/jumbo frame related, just plain e1000/i350 IPv6 regression.
>
>
> Happy to test anything, can test-drive swiftly but without further diag
> during work days.
>
> Thanks,
> -harry

Thanks for the report.  I added Franco and Stephen to the cc for visibility.

Nothing is immediately jumping out at me, in a private email Harry
tried not setting 'rxcsum |= E1000_RXCSUM_TUOFL | E1000_RXCSUM_IPOFL'
which was an intentional behavior change but it did not improve this
IPv6 use.

I will need to do some document reading.

Regards,
Kevin

Re: igb(4) and VLAN issue?

[Kevin Bowling]

Franco,

I think I found it: https://reviews.freebsd.org/D32087

Regards,
Kevin

On Tue, Aug 3, 2021 at 8:50 AM Kevin Bowling  wrote:
>
> On Tue, Aug 3, 2021 at 8:27 AM Franco Fichtner  wrote:
> >
> > Hi Kevin,
> >
> > [RESENT TO MAILING LIST AS SUBSCRIBER]
> >
> > > On 2. Aug 2021, at 7:51 PM, Kevin Bowling  
> > > wrote:
> > >
> > > I caught wind that an igb(4) commit I've done to main and that has
> > > been in stable/12 for a few months seems to be causing a regression on
> > > opnsense.  The commit in question is
> > > https://cgit.freebsd.org/src/commit/?id=eea55de7b10808b86277d7fdbed2d05d3c6db1b2
> > >
> > > The report is at:
> > > https://forum.opnsense.org/index.php?topic=23867.0
> >
> > Looks like I spoke to soon earlier.  This is a weird one for sure.  :)
> >
> > So first of all this causes an ifconfig hang for VLAN/LAGG combo creation,
> > but later reports were coming in about ahci errors and cam timeouts.
> > Some reported the instabilities start with using netmap, but later others
> > confirmed the same for high load scenarios without netmap in use.
> >
> > The does not appear to happen when MSIX is disabled, e.g.:
> >
> > # sysctl -a | grep dev.igb | grep msix
> > dev.igb.5.iflib.disable_msix: 1
> > dev.igb.4.iflib.disable_msix: 1
> > dev.igb.3.iflib.disable_msix: 1
> > dev.igb.2.iflib.disable_msix: 1
> > dev.igb.1.iflib.disable_msix: 1
> > dev.igb.0.iflib.disable_msix: 1
> >
> > What's also being linked to this is some form of softraid misbehaving
> > and the general tendency for cheaper hardware with particular igb
> > chipsets.
>
> Hmm, there is so much that /could/ be going on it's not easy to
> pinpoint anything yet.  If nothing jumps out after getting more data
> it may be worth mitigating in your build that way and retrying once
> you have updated to FreeBSD 13.
>
> > > I haven't heard of this issue elsewhere and cannot replicate it on my
> > > I210s running main.  I've gone over the code changes line by line
> > > several times and verified all the logic and register writes and it
> > > all looks correct to my understanding.  The only hypothesis I have at
> > > the moment is it may be some subtle timing issue since VLAN changes
> > > unnecessarily restart the interface on e1000 until I push in a work in
> > > progress to stop doing that.
> >
> > I also have no way of reproducing this locally, but the community is
> > probably willing to give any kernel change a try that would address
> > the problem without havinbg to back out the commit in question.
>
> I need some more info before making any changes.  A full dmesg of the
> older working version and a (partial?) dmesg of the broken would be
> another useful data point to start out with, let's see if there is
> something going on during MSI-X vector allocation etc.
>
> > > I'd like to see the output of all the processes or at least the
> > > process configuring the VLANs to see where it is stuck.  Franco, do
> > > you have the ability to 'control+t' there or otherwise set up a break
> > > into a debugger?  Stacktraces would be a great start but a core and a
> > > kernel may be necessary if it isn't obvious.
> >
> > Let me see if I can deliver on this easily.
> >
> >
> > Cheers,
> > Franco
> >

Re: igb(4) and VLAN issue?

[Kevin Bowling]

On Tue, Aug 3, 2021 at 8:27 AM Franco Fichtner  wrote:
>
> Hi Kevin,
>
> [RESENT TO MAILING LIST AS SUBSCRIBER]
>
> > On 2. Aug 2021, at 7:51 PM, Kevin Bowling  wrote:
> >
> > I caught wind that an igb(4) commit I've done to main and that has
> > been in stable/12 for a few months seems to be causing a regression on
> > opnsense.  The commit in question is
> > https://cgit.freebsd.org/src/commit/?id=eea55de7b10808b86277d7fdbed2d05d3c6db1b2
> >
> > The report is at:
> > https://forum.opnsense.org/index.php?topic=23867.0
>
> Looks like I spoke to soon earlier.  This is a weird one for sure.  :)
>
> So first of all this causes an ifconfig hang for VLAN/LAGG combo creation,
> but later reports were coming in about ahci errors and cam timeouts.
> Some reported the instabilities start with using netmap, but later others
> confirmed the same for high load scenarios without netmap in use.
>
> The does not appear to happen when MSIX is disabled, e.g.:
>
> # sysctl -a | grep dev.igb | grep msix
> dev.igb.5.iflib.disable_msix: 1
> dev.igb.4.iflib.disable_msix: 1
> dev.igb.3.iflib.disable_msix: 1
> dev.igb.2.iflib.disable_msix: 1
> dev.igb.1.iflib.disable_msix: 1
> dev.igb.0.iflib.disable_msix: 1
>
> What's also being linked to this is some form of softraid misbehaving
> and the general tendency for cheaper hardware with particular igb
> chipsets.

Hmm, there is so much that /could/ be going on it's not easy to
pinpoint anything yet.  If nothing jumps out after getting more data
it may be worth mitigating in your build that way and retrying once
you have updated to FreeBSD 13.

> > I haven't heard of this issue elsewhere and cannot replicate it on my
> > I210s running main.  I've gone over the code changes line by line
> > several times and verified all the logic and register writes and it
> > all looks correct to my understanding.  The only hypothesis I have at
> > the moment is it may be some subtle timing issue since VLAN changes
> > unnecessarily restart the interface on e1000 until I push in a work in
> > progress to stop doing that.
>
> I also have no way of reproducing this locally, but the community is
> probably willing to give any kernel change a try that would address
> the problem without havinbg to back out the commit in question.

I need some more info before making any changes.  A full dmesg of the
older working version and a (partial?) dmesg of the broken would be
another useful data point to start out with, let's see if there is
something going on during MSI-X vector allocation etc.

> > I'd like to see the output of all the processes or at least the
> > process configuring the VLANs to see where it is stuck.  Franco, do
> > you have the ability to 'control+t' there or otherwise set up a break
> > into a debugger?  Stacktraces would be a great start but a core and a
> > kernel may be necessary if it isn't obvious.
>
> Let me see if I can deliver on this easily.
>
>
> Cheers,
> Franco
>

igb(4) and VLAN issue?

[Kevin Bowling]

Hi,

I caught wind that an igb(4) commit I've done to main and that has
been in stable/12 for a few months seems to be causing a regression on
opnsense.  The commit in question is
https://cgit.freebsd.org/src/commit/?id=eea55de7b10808b86277d7fdbed2d05d3c6db1b2

The report is at:
https://forum.opnsense.org/index.php?topic=23867.0

I haven't heard of this issue elsewhere and cannot replicate it on my
I210s running main.  I've gone over the code changes line by line
several times and verified all the logic and register writes and it
all looks correct to my understanding.  The only hypothesis I have at
the moment is it may be some subtle timing issue since VLAN changes
unnecessarily restart the interface on e1000 until I push in a work in
progress to stop doing that.

I'd like to see the output of all the processes or at least the
process configuring the VLANs to see where it is stuck.  Franco, do
you have the ability to 'control+t' there or otherwise set up a break
into a debugger?  Stacktraces would be a great start but a core and a
kernel may be necessary if it isn't obvious.

Regards,
Kevin

freebsd-net call

[Kevin Bowling]

Hi,

I would like to gauge interest in holding a regular freebsd-net video
call.  This would cover primarily device drivers, pseudo interfaces,
and the programming interfaces like iflib and ifnet.  There is a
separate transport meeting so it would be complementary to that.

Please send me a note if you are interested and what timezone you are in.

Regards,
Kevin

Re: Vector Packet Processing (VPP) portability on FreeBSD

[Kevin Bowling]

The one other thing I want to mention, what this means in effect is every
que ends up limited by EITR on ixgbe (around 30kps with the default
settings) whether it’s a TX or RX workload.  This ends up working ok if you
have sufficient CPU but seems awkward.  On the TX workload we should need a
magnitude less interrupts to do 10g. There was some work to adapt AIM to
this new combined handler but it is not properly tuned and I’m not sure it
should consider TX at all.

Regards,
Kevin

On Mon, May 24, 2021 at 11:16 PM Kevin Bowling 
wrote:

> I don't fully understand the issue, but in iflib_fast_intr_rxtx
> https://cgit.freebsd.org/src/tree/sys/net/iflib.c#n1581 it seems like
> we end up re-enabling interrupts per course instead of only handling
> spurious cases or some low water threshold (which seems like it would
> be tricky to do here).  The idea is we want to pump interrupts by
> disabling them in the msix_que handler, and then wait to re-enable
> only when we have more work to do in the ift_task grouptask.
>
> It was a lot easier to reason about this with separate TX and RX
> interrupts.  Doing the combined TXRX is definitely a win in terms of
> reducing msi-x vector usage (which is important in a lot of FreeBSD
> use cases), but it's tricky to understand.
>
> My time has been sucked away due to work, so I haven't been looking at
> this problem to the depth I want to.  I'd be interested in discussing
> it further with anyone that is interested in it.
>
> Regards,
> Kevin
>
> On Tue, May 18, 2021 at 2:11 PM Vincenzo Maffione 
> wrote:
> >
> >
> >
> > Il giorno mar 18 mag 2021 alle ore 09:32 Kevin Bowling <
> kevin.bowl...@kev009.com> ha scritto:
> >>
> >>
> >>
> >> On Mon, May 17, 2021 at 10:20 AM Marko Zec  wrote:
> >>>
> >>> On Mon, 17 May 2021 09:53:25 +
> >>> Francois ten Krooden  wrote:
> >>>
> >>> > On 2021/05/16 09:22, Vincenzo Maffione wrote:
> >>> >
> >>> > >
> >>> > > Hi,
> >>> > >   Yes, you are not using emulated netmap mode.
> >>> > >
> >>> > >   In the test setup depicted here
> >>> > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
> >>> > > interfaces#test-setup
> >>> > > I think you should really try to replace VPP with the netmap
> >>> > > "bridge" application (tools/tools/netmap/bridge.c), and see what
> >>> > > numbers you get.
> >>> > >
> >>> > > You would run the application this way
> >>> > > # bridge -i ix0 -i ix1
> >>> > > and this will forward any traffic between ix0 and ix1 (in both
> >>> > > directions).
> >>> > >
> >>> > > These numbers would give you a better idea of where to look next
> >>> > > (e.g. VPP code improvements or system tuning such as NIC
> >>> > > interrupts, CPU binding, etc.).
> >>> >
> >>> > Thank you for the suggestion.
> >>> > I did run a test with the bridge this morning, and updated the
> >>> > results as well. +-+--+
> >>> > | Packet Size | Throughput (pps) |
> >>> > +-+--+
> >>> > |   64 bytes  |7.197 Mpps|
> >>> > |  128 bytes  |7.638 Mpps|
> >>> > |  512 bytes  |2.358 Mpps|
> >>> > | 1280 bytes  |  964.915 kpps|
> >>> > | 1518 bytes  |  815.239 kpps|
> >>> > +-+--+
> >>>
> >>> I assume you're on 13.0 where netmap throughput is lower compared to
> >>> 11.x due to migration of most drivers to iflib (apparently increased
> >>> overhead) and different driver defaults.  On 11.x I could move 10G line
> >>> rate from one ix to another at low CPU freqs, where on 13.x the CPU
> >>> must be set to max speed, and still can't do 14.88 Mpps.
> >>
> >>
> >> I believe this issue is in the combined txrx interrupt filter.  It is
> causing a bunch of unnecessary tx re-arms.
> >
> >
> > Could you please elaborate on that?
> >
> > TX completion is indeed the one thing that changed considerably with the
> porting to iflib. And this could be a major contributor to the performance
> drop.
> > My understanding is that TX interrupts are not really used anymore on
> multi-gigabit NICs such as ix or ixl. Instead, "softirqs" are used, meaning
> that a t

Re: Vector Packet Processing (VPP) portability on FreeBSD

[Kevin Bowling]

I don't fully understand the issue, but in iflib_fast_intr_rxtx
https://cgit.freebsd.org/src/tree/sys/net/iflib.c#n1581 it seems like
we end up re-enabling interrupts per course instead of only handling
spurious cases or some low water threshold (which seems like it would
be tricky to do here).  The idea is we want to pump interrupts by
disabling them in the msix_que handler, and then wait to re-enable
only when we have more work to do in the ift_task grouptask.

It was a lot easier to reason about this with separate TX and RX
interrupts.  Doing the combined TXRX is definitely a win in terms of
reducing msi-x vector usage (which is important in a lot of FreeBSD
use cases), but it's tricky to understand.

My time has been sucked away due to work, so I haven't been looking at
this problem to the depth I want to.  I'd be interested in discussing
it further with anyone that is interested in it.

Regards,
Kevin

On Tue, May 18, 2021 at 2:11 PM Vincenzo Maffione  wrote:
>
>
>
> Il giorno mar 18 mag 2021 alle ore 09:32 Kevin Bowling 
>  ha scritto:
>>
>>
>>
>> On Mon, May 17, 2021 at 10:20 AM Marko Zec  wrote:
>>>
>>> On Mon, 17 May 2021 09:53:25 +
>>> Francois ten Krooden  wrote:
>>>
>>> > On 2021/05/16 09:22, Vincenzo Maffione wrote:
>>> >
>>> > >
>>> > > Hi,
>>> > >   Yes, you are not using emulated netmap mode.
>>> > >
>>> > >   In the test setup depicted here
>>> > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
>>> > > interfaces#test-setup
>>> > > I think you should really try to replace VPP with the netmap
>>> > > "bridge" application (tools/tools/netmap/bridge.c), and see what
>>> > > numbers you get.
>>> > >
>>> > > You would run the application this way
>>> > > # bridge -i ix0 -i ix1
>>> > > and this will forward any traffic between ix0 and ix1 (in both
>>> > > directions).
>>> > >
>>> > > These numbers would give you a better idea of where to look next
>>> > > (e.g. VPP code improvements or system tuning such as NIC
>>> > > interrupts, CPU binding, etc.).
>>> >
>>> > Thank you for the suggestion.
>>> > I did run a test with the bridge this morning, and updated the
>>> > results as well. +-+--+
>>> > | Packet Size | Throughput (pps) |
>>> > +-+--+
>>> > |   64 bytes  |7.197 Mpps|
>>> > |  128 bytes  |7.638 Mpps|
>>> > |  512 bytes  |2.358 Mpps|
>>> > | 1280 bytes  |  964.915 kpps|
>>> > | 1518 bytes  |  815.239 kpps|
>>> > +-+--+
>>>
>>> I assume you're on 13.0 where netmap throughput is lower compared to
>>> 11.x due to migration of most drivers to iflib (apparently increased
>>> overhead) and different driver defaults.  On 11.x I could move 10G line
>>> rate from one ix to another at low CPU freqs, where on 13.x the CPU
>>> must be set to max speed, and still can't do 14.88 Mpps.
>>
>>
>> I believe this issue is in the combined txrx interrupt filter.  It is 
>> causing a bunch of unnecessary tx re-arms.
>
>
> Could you please elaborate on that?
>
> TX completion is indeed the one thing that changed considerably with the 
> porting to iflib. And this could be a major contributor to the performance 
> drop.
> My understanding is that TX interrupts are not really used anymore on 
> multi-gigabit NICs such as ix or ixl. Instead, "softirqs" are used, meaning 
> that a timer is used to perform TX completion. I don't know what the 
> motivations were for this design decision.
> I had to decrease the timer period to 90us to ensure timely completion (see 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652). However, the timer 
> period is currently not adaptive.
>
>
>>
>>
>>>
>>> #1 thing which changed: default # of packets per ring dropped down from
>>> 2048 (11.x) to 1024 (13.x).  Try changing this in /boot/loader.conf:
>>>
>>> dev.ixl.0.iflib.override_nrxds=2048
>>> dev.ixl.0.iflib.override_ntxds=2048
>>> dev.ixl.1.iflib.override_nrxds=2048
>>> dev.ixl.1.iflib.override_ntxds=2048
>>> etc.
>>>
>>> For me this increases the throughput of
>>> bridge -i netmap:ixl0 -i netmap:ixl1
>>> from 9.3 Mpps to 11.4 Mpps
>>>
>>> #2: default interrupt moderation delays 

Re: Vector Packet Processing (VPP) portability on FreeBSD

[Kevin Bowling]

On Mon, May 17, 2021 at 10:20 AM Marko Zec  wrote:

> On Mon, 17 May 2021 09:53:25 +
> Francois ten Krooden  wrote:
>
> > On 2021/05/16 09:22, Vincenzo Maffione wrote:
> >
> > >
> > > Hi,
> > >   Yes, you are not using emulated netmap mode.
> > >
> > >   In the test setup depicted here
> > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
> > > interfaces#test-setup
> > > I think you should really try to replace VPP with the netmap
> > > "bridge" application (tools/tools/netmap/bridge.c), and see what
> > > numbers you get.
> > >
> > > You would run the application this way
> > > # bridge -i ix0 -i ix1
> > > and this will forward any traffic between ix0 and ix1 (in both
> > > directions).
> > >
> > > These numbers would give you a better idea of where to look next
> > > (e.g. VPP code improvements or system tuning such as NIC
> > > interrupts, CPU binding, etc.).
> >
> > Thank you for the suggestion.
> > I did run a test with the bridge this morning, and updated the
> > results as well. +-+--+
> > | Packet Size | Throughput (pps) |
> > +-+--+
> > |   64 bytes  |7.197 Mpps|
> > |  128 bytes  |7.638 Mpps|
> > |  512 bytes  |2.358 Mpps|
> > | 1280 bytes  |  964.915 kpps|
> > | 1518 bytes  |  815.239 kpps|
> > +-+--+
>
> I assume you're on 13.0 where netmap throughput is lower compared to
> 11.x due to migration of most drivers to iflib (apparently increased
> overhead) and different driver defaults.  On 11.x I could move 10G line
> rate from one ix to another at low CPU freqs, where on 13.x the CPU
> must be set to max speed, and still can't do 14.88 Mpps.
>

I believe this issue is in the combined txrx interrupt filter.  It is
causing a bunch of unnecessary tx re-arms.


> #1 thing which changed: default # of packets per ring dropped down from
> 2048 (11.x) to 1024 (13.x).  Try changing this in /boot/loader.conf:
>
> dev.ixl.0.iflib.override_nrxds=2048
> dev.ixl.0.iflib.override_ntxds=2048
> dev.ixl.1.iflib.override_nrxds=2048
> dev.ixl.1.iflib.override_ntxds=2048
> etc.
>
> For me this increases the throughput of
> bridge -i netmap:ixl0 -i netmap:ixl1
> from 9.3 Mpps to 11.4 Mpps
>
> #2: default interrupt moderation delays seem to be too long.  Combined
> with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62
> (default) to 40 increases the throughput further from 11.4 to 14.5 Mpps
>
> Hope this helps,
>
> Marko
>
>
> > Besides for the 64-byte and 128-byte packets the other sizes where
> > matching the maximum rates possible on 10Gbps. This was when the
> > bridge application was running on a single core, and the cpu core was
> > maxing out at a 100%.
> >
> > I think there might be a bit of system tuning needed, but I suspect
> > most of the improvement would be needed in VPP.
> >
> > Regards
> > Francois
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Client Networking Issues / NIC Lab

[Kevin Bowling]

On Fri, Apr 23, 2021 at 6:19 AM Rick Macklem  wrote:
>
> Kyle Evans wrote:
> >On Fri, Apr 23, 2021 at 12:22 AM Kevin Bowling  
> >wrote:
> >>
> >> Greetings,
> >>
> >> [... snip ...]
> >>
> >> Tehuti Networks seems to have gone out of business.  Probably not
> >> worth worrying about.
> >>
> >
> >That's unfortunate. I had a box of their 10G NICs and I got them to
> >put a driver up for review[0][1], but they weren't very responsive and
> >the existing codebase was in pretty rough shape.
> >
> >Beyond that, your #3 seems to be the most appealing. #2 could probably
> >work in the mid-to-long term, but we'd likely be better off
> >bootstrapping interest with solid community-supported drivers then
> >reaching out to vendors once we can demonstrate that plan field of
> >dreams can work and drive some substantial amount of business.
>
> I'll admit to knowing nothing about it, but is using the linuxKPI
> to port Linux drivers into FreeBSD feasible?

Hi Rick,

I did consider this but do not think it makes sense for PCI Ethernet
NIC drivers.  I will explain my judgement for consideration.  In
complex systems such as an Ethernet driver, there are intrinsic and
extrinsic complexity.  The intrinsic properties of an Ethernet driver
are small enough that one person can understand them.  So we spend a
lot of time fighting against extrinsic problems that I outlined in my
email. Put in simpler terms, an iflib driver can be written by one
person and there are a number of people that are good at this in the
community.  The intrinsic complexity of the LKPI on top of an Ethernet
driver, as well as some license and social problems people have with
LKPI lead it to be a worse fit.

If you apply this to drm+i915 etc it is illuminating why the Linux KPI
is the right approach.  The intrinsic properties of the graphics stack
are beyond time and practicality for most in the community, the
graphics drivers have become labyrinths that most kernel devs don't
have internal knowledge of, rival the size of the rest of the kernel,
and keeping up is easier if internal code changes can be kept to a
minimum.

> Obviously, given the size of the Linux community, it seems
> more likely that it will have a driver that handles many chip
> variants, plus updates for newer chips, I think.

I would agree that Linux has a much better Realtek driver.  I am
familiar with the Linux e1000 series for instance, and although they
tend to have most the workarounds the quality is a lot lower than most
users realize.

> I do agree that having drivers that at least work for the
> basics (maybe no Netmap, TSO, or similar) for the
> commodity chips would make it easier for new adopters
> of FreeBSD. (I avoid the problem by finding old, used
> hardware. The variants of Intel PRO1000 and re chips I
> have work fine with the drivers in FreeBSD13/14.;-)

Having good inbox network drivers is a way for FreeBSD to
differentiate itself.  I like nice drivers like cxgbe(4), it is a
great piece of engineering and to me even artful.  Consider some cxgbe
so you can test high speeds :)

> Oh, and if TSO support is questionable, I think it would be
> better to leave it disabled and at least generate a warning
> when someone enables it, if it can be enabled at all.

I would like to preserve and correct TSO and other offloads as much as
possible.  There are consequences to half assing it such as burning
more electricity than necessary and causing unnecessary HW
upgrade/replacement.  Of course, where we can't deliver, we should
limit the feature set to known good ones.  Striking this balance will
require more feedback from the community, with faster turnaround time
on PRs.

> Good luck with it, rick
>
> Thanks,
>
> Kyle Evans
>
> [0] https://reviews.freebsd.org/D18856
> [1] https://reviews.freebsd.org/D19433
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Client Networking Issues / NIC Lab

[Kevin Bowling]

Greetings,

I have been looking into client networking issues in FreeBSD lately.
To summarize the situation, common NICs like Intel gigabit (e1000 aka
lem(4)/em(4)/igb(4)), Realtek (re(4)), Aquantia, and Tehuti Networks
are unmaintained or not present on FreeBSD.  The purpose of this
thread is to gauge whether that matters, and if it does what to do.  I
believe it is important because we are losing out on a pipeline of new
contributors by not supporting client hardware well.  We risk losing
NAS, firewall, and other embedded users which may not be large enough
to negotiate with these vendors for support or have the volume to do
custom BOMs to avoid risky parts.  My opinion has been developed after
researching the drivers, Bugzilla, and various internet forums where
end users exchange advice or ask for help where FreeBSD is the
underlying cause.

e1000 is in the best shape, with recent vendor involvement, but covers
20 years of silicon with over 100 chipsets (of which at least 60 are
significant variations).  Datasheets are readily available for most of
them, as well as "specification updates" which list errata.  There are
chipsets which have been completely broken for several years.  More
common, there are cases that lead to user frustration, including with
the most recent hardware.  All of the silicon tends to have
significant bugs around PCIe, TSO, NC-SI (IPMI sideband), arbitration
conflicts with ME and more.  Intel doesn't patch the microcode on
these, but many of the issues can be worked around in software.
Performing an audit of the driver will take quite a while, and making
and testing changes gives me concern.  When we (my previous employer
and team) converted these drivers to iflib, we fixed some of the
common cases for PCIe and TSO issues but only had a handful of chips
to test against, so the driver works better for some and worse or not
at all for others.  I have started fixing some of the bugs in
Bugzilla, but I only have a few e1000 variants on hand to test, and I
have an unrelated full time job so this is just occupying limited
spare time as a hobby.

re(4) is in pretty abhorrent state.  All of these chips require
runtime patching of the phy (which I believe is a DSP algorithm that
gets improved over time) and mcu code.  That is totally absent in
FreeBSD.  A vendor driver exists in net/realtek-re-kmod which contains
the fixups and works alright for many users.  This driver cannot be
imported into FreeBSD as is.  There is a strange use of the C
PreProcessor which blows up compile time and driver size needlessly.
The out of tree driver has a different set of supported adapters, so
some kind of meld is necessary.  Realtek does not provide public chip
documentation, I am trying to see if they will grant NDA access to
contributors.

Aquantia has an out of tree driver in net/aquantia-atlantic-kmod.  The
code is not currently in a place where I'd like to see it in the tree.
I am not really sure how common these are, the company was acquired by
Marvell which is still producing them as a client networking option
while they have other IP for higher end/speed.

Tehuti Networks seems to have gone out of business.  Probably not
worth worrying about.

1) Do nothing.  This situation has gone on for a while.  Users are
somewhat accustomed to purchasing FreeBSD-specific hardware for things
like SOHO gateways and NAS.  A lot of people just revert back to Linux
for client use.  OpenBSD seems to have more active contribution around
this kind of thing and works better for common cases so that may be
another exit ramp.

2) Quantify usage data and beg the vendors for help.  This might work
for Intel, however these devices have transferred to a client team at
intel that does not plan to support FreeBSD, and intel does not keep
test systems around long enough to meet FreeBSD user's needs. Realtek
is a similar story, I am unsure how long they hold on to test systems
and would probably need technical guidance to work with the FreeBSD
community.  Unsure about Marvell, I've never worked with them.

3) Build a NIC lab and focus on building community support.  It would
also give the vendors a place to test hardware their labs have purged
(due to IT asset management policies or other bureaucratic blunders).
Set some boundaries like a 15 year window of chipsets which should
cover practical embedded use cases.  There are backplane systems
and/or external PCI(e) expansion systems that could be assembled to
house a large number of NICs.  It would probably be cheaper than this,
but say a budget of $15000USD is enough to purchase some expansions, a
couple managed switches, and a few dozen common NICs.  Community
members may also send in NICs they wish to see supported or undergo
testing.  For this to work out long term, there needs to be a quorum
of people interested in collaborating on the issue.  There are some
risks around simply setting this up, depending on the configuration,
the bus topology may introduce problems 

Re: All transmits on txq0 on an ix interface - why no balancing?

[Kevin Bowling]

On Tue, Apr 13, 2021 at 5:37 PM Phil Rosenthal  wrote:
>
> > On Apr 13, 2021, at 8:22 PM, Kevin Bowling  wrote:
> >
> > On Tue, Apr 13, 2021 at 5:11 PM Phil Rosenthal  wrote:
> >>
> >>> On Apr 13, 2021, at 8:07 PM, Olivier Cochard-Labbé  
> >>> wrote:
> >>>
> >>> Are you exiting through a tunnel interface (GRE, GIF, PPPoE, IPsec, 
> >>> OpenVPN, etc.) ?
> >> No.
> >>
> >> I am running PF/Altq for NAT and Traffic shaping.
> >
> > ALTQ is the problem in this situation.  Try without it and see if you
> > get proper distribution.
>
> Yep, proper distribution with altq disabled.
>
> That's pretty unfortunate, though ... It means that I'm forced to deal with 
> the Comcast's buffer management which allows for more buffer bloat and is 
> less deterministic about what packets get dropped.
>
> I'm assuming that this won't get fixed due to PF generally not being 
> maintained these days -- is there any other solution for shaping with active 
> queue management (preferably RED) on FreeBSD to use instead of altq? Dummynet 
> obviously has problems of it's own.

pf in FreeBSD is pretty well maintained.

altq currently has no maintainer (at least that I am aware of).  It's
a difficult problem, altq needs to "learn" a lot about multithreading.
I'm not aware of anyone working on it, although Patrick Kelsey
(pkelsey@) may be worth a ping (IIRC he might have looked at this
problem a bit, I'd bet he could do it if he has time and some entity
can sponsor the work).

> In any event - thanks for your help.
> -Phil
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: All transmits on txq0 on an ix interface - why no balancing?

[Kevin Bowling]

On Tue, Apr 13, 2021 at 5:11 PM Phil Rosenthal  wrote:
>
> > On Apr 13, 2021, at 8:07 PM, Olivier Cochard-Labbé  
> > wrote:
> >
> > Are you exiting through a tunnel interface (GRE, GIF, PPPoE, IPsec, 
> > OpenVPN, etc.) ?
> No.
>
> I am running PF/Altq for NAT and Traffic shaping.

ALTQ is the problem in this situation.  Try without it and see if you
get proper distribution.

> One port is using a Multirate 2.5G SFP+ (linked at 2.5G) to connect to a 
> cablemodem
>
> The other port is linked to a 10G DAC to a Juniper switch
>
> The goal is to be able to get the 1.44 Gbps provisioned speeds from Comcast, 
> but it seems that bursts over 1.3Gbps cause a small amount of packet loss 
> which causes speeds to drop back down.
>
> -Phil
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Which cpu/mainboard for fast routing (bgp, full tables) ?

[Kevin Bowling]

That class of processor has fairly limited memory bandwidth.  An E5 v3 or
greater should get you what you want, although finding a system that makes
good use of available PCIe lanes with a single socket configuration can
sometimes be maddening.  AMD may have a variety of nice parts for this
application, although I don’t have any personal experience with routing on
such hardware.  Probably equally important is the NICs, I’d go with Chelsio
T5 or Mellanox ConnectX 4 Lx or greater.

On Sat, Mar 27, 2021 at 1:08 PM Kurt Jaeger  wrote:

> Hi!
>
> We currently operate routers (FreeBSD 12.x, frr7) with
> Xeon(R) CPU E3-1230 v6 @ 3.50GHz CPUs and 10g links.
> They get to around 5-6 gbit/s throughput.
>
> What kind of hardware can you all suggest, if we stay
> in the generic PC area, to improve the routing throughput ?
>
> --
> p...@freebsd.org +49 171 3101372  Now what ?
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

tcp-testsuite into src?

[Kevin Bowling]

Hi,

I was talking with gnn and kevans on IRC about the tcp testsuite
(https://github.com/freebsd-net/tcp-testsuite).

Currently we maintain this in ports, although the way the port is set
up doesn't make a lot of sense because the tests are stack specific
and we don't account for different FreeBSD versions let alone vendor
trees.  It seems reasonable to me to pull the tests themselves (i.e.
https://github.com/freebsd-net/tcp-testsuite) into src where they can
follow along with the tree they are running on, and provide vendors a
natural point of extension.

/usr/tests has some existing examples of relying on out of tree
binaries to run so I am not convinced we need to import packetdrill
itself but I don't have a strong preference.  tuexen, do you have any
preference?

Regards,
Kevin
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: TCP Congestion Control

[Kevin Bowling]

The per socket method is used at a large commercial CDN

On Thu, Oct 24, 2019 at 12:30 AM vm finance  wrote:

> Ok - I  see there is a socket option to pick a different cc per-socket
> basis.
> Any experiences on loading / using different cc per socket...does it work
> seamlessly?
>
> Thanks!
>
> On Wed, Oct 23, 2019 at 11:19 PM Kevin Oberman 
> wrote:
>
> > Have you loaded kernel modules for other algorithms? I believe only
> > newreno is in the default kernel. "man 4 mod_cc" for available modules
> and
> > other information.
> > --
> > Kevin Oberman, Part time kid herder and retired Network Engineer
> > E-mail: rkober...@gmail.com
> > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
> >
> >
> > On Wed, Oct 23, 2019 at 11:04 PM vm finance 
> wrote:
> >
> >> Hi,
> >>
> >> We can set per-socket congestion control under Linux, but not under
> >> FreeBSD
> >> (12.0).
> >> The current available and allowed is only newReno:
> >> net.inet.tcp.cc.available: newreno
> >> net.inet.tcp.cc.algorithm: newreno
> >>
> >> Any thoughts on why FreeBSD chose not to allow different cc to be set
> per
> >> socket?
> >> AFAIK, it would get complicated to have different sessions having
> >> different
> >> congestion algos.
> >>
> >> Thanks for any pointers!
> >> ___
> >> freebsd-net@freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> >>
> >
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Issues with TCP Timestamps allocation

[Kevin Bowling]

Any knowledge of the endpoints, Linux boxes misconfigured with
tcp_tw_recycle?

On Wed, Jul 17, 2019 at 5:42 AM Michael Tuexen  wrote:

> > On 17. Jul 2019, at 14:32, Vitalij Satanivskij  wrote:
> >
> > Hmm, looks like with some host's work but not with another
> >
> > Wed/17.07:/home/satan
> > hell:-1522/15:28>curl https://volia.com > /dev/null
> >  % Total% Received % Xferd  Average Speed   TimeTime Time
> Current
> > Dload  Upload   Total   SpentLeft
> Speed
> > 100 415190 415190 0   137k  0 --:--:-- --:--:--
> --:--:--  137k
> > Wed/17.07:/home/satan
> > hell:-1523/15:28>curl https://volia.com > /dev/null
> >  % Total% Received % Xferd  Average Speed   TimeTime Time
> Current
> > Dload  Upload   Total   SpentLeft
> Speed
> >  0 00 00 0  0  0 --:--:--  0:00:53 --:--:--
>0^C
> > Wed/17.07:/home/satan
> > hell:-1524/15:29>sysctl net.inet.tcp.rexmit_drop_options
> > net.inet.tcp.rexmit_drop_options: 1
> OK, I can confirm that for https://volia.com only a timeout helps.
>
> What I observed for now is that for the "blocking" to occur is it crucial
> that
> the server sends the FIN and therefore goes into the TIMEWAIT state. The
> timeout
> seems to be 60 seconds.
> The blocking is also not limited to a single server port.
>
> I'm not sure yet whether it is a broken end point or a broken middle box.
>
> Best regards
> Michael
> >
> > But
> >
> > MT> Interesting. It works for me:
> > MT>
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0  33637  0 --:--:-- --:--:--
> --:--:-- 33575
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0   4834  0 --:--:--  0:00:03
> --:--:--  4833
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0  35813  0 --:--:-- --:--:--
> --:--:-- 35813
> > MT> tuexen@head:~ % time curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0  48320  0 --:--:-- --:--:--
> --:--:-- 48320
> > MT> 0.012u 0.031s 0:00.39 10.2%   140+245k 0+0io 0pf+0w
> > MT> tuexen@head:~ % time curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0   4592  0 --:--:--  0:00:03
> --:--:--  4591
> > MT> 0.031u 0.010s 0:03.99 1.0%80+140k 0+0io 0pf+0w
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0  37815  0 --:--:-- --:--:--
> --:--:-- 37737
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0  27261  0 --:--:-- --:--:--
> --:--:-- 27220
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0   4533  0 --:--:--  0:00:04
> --:--:--  4533
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0  48320  0 --:--:-- --:--:--
> --:--:-- 48192
> > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null
> > MT>   % Total% Received % Xferd  Average Speed   TimeTime
>  Time  Current
> > MT>  Dload  Upload   Total   Spent
> Left  Speed
> > MT> 100 182650 182650 0   4746  0 --:--:--  0:00:03
> --:--:--  4745
> > MT> tuexen@head:~ % curl https://vitagramma.com > 

Re: igb: device_attach: igb1 attach returned 6

[Kevin Bowling]

I looked through the phy common code differences with linux master and
nothing jumped out at me that would impact this i350 setup.  It would
still be interesting to run that code and see if it works or not when
you get time.

It'd be best to also loop intel in directly since you are a customer
and it appears to affect common code or hardware issue.

Regards,

On Thu, Aug 23, 2018 at 12:51 AM, Roman Bogorodskiy
 wrote:
>   Kevin Bowling wrote:
>
>> Well I took a look at the phy code and actually don't see that we
>> abosrbed those commits.  I'll try and audit e1000 vs linux in the next
>> 24 hours but in the mean time if you could confirm linux-next or the
>> like fix or don't fix the phy we can at least know if this is the
>> right path or if we need to ask Intel networking for help.
>
> Yeah, I see the code that resets PHY in e1000_82575.c,
> e1000_setup_copper_link_82575(), but I don't see this being called at
> all in my dmesg output, so I guess removing that code will not change
> anything.
>
> I wasn't able to find a code similar to igb_write_phy_reg_82580(hw,
> I347AT4_PAGE_SELECT, 0); from that net-next patch.
>
> There's a code in e1000_phy.c e1000_get_cable_length_m88_gen2() that
> writes to I347AT4_PAGE_SELECT (it uses 0x07 or 0x05 instead of 0 though), but 
> I
> don't see it in dmesg either...
>
> I'll try to find a spare drive to install Linux, but I'm not sure how
> soon I'll be able to do that...
>
>> Regards,
>>
>> On Thu, Aug 23, 2018 at 12:19 AM, Kevin Bowling
>>  wrote:
>> > Great thanks for the info about it not working on Ubuntu too, that
>> > narrows down the problem space considerably.
>> >
>> > Can you try that patch on FreeBSD -CURRENT?  It should apply with
>> > minimal changes the phy code is shared by multiple OSes.  If it works
>> > we can commit.
>> >
>> > Regards,
>> >
>> > On Thu, Aug 23, 2018 at 12:16 AM, Roman Bogorodskiy
>> >  wrote:
>> >> I haven't used this hardware before, so I don't if it ever worked. I
>> >> only tried 11.2 and it doesn't work. I've also tried Ubuntu 18.04 and it
>> >> doesn't work as well. Ubuntu prints the following error:
>> >>
>> >> [  343.053031] igb: Intel(R) Gigabit Ethernet Network Driver - version 
>> >> 5.4.0-k
>> >> [  343.053032] igb: Copyright (c) 2007-2014 Intel Corporation.
>> >> [  343.415346] igb :01:00.0: added PHC on eth0
>> >> [  343.415349] igb :01:00.0: Intel(R) Gigabit Ethernet Network 
>> >> Connection
>> >> [  343.415352] igb :01:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 
>> >> 00:12:c0:00:e9:14
>> >> [  343.415427] igb :01:00.0: eth0: PBA No: 106100-000
>> >> [  343.415430] igb :01:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 
>> >> tx queue(s)
>> >> [  343.416422] igb :01:00.0 enp1s0f0: renamed from eth0
>> >> [  343.416827] igb: probe of :01:00.1 failed with error -3
>> >> [  343.485242] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
>> >> [  343.777894] igb :01:00.2: added PHC on eth0
>> >> [  343.777897] igb :01:00.2: Intel(R) Gigabit Ethernet Network 
>> >> Connection
>> >> [  343.777900] igb :01:00.2: eth0: (PCIe:5.0Gb/s:Width x4) 
>> >> 00:12:c0:00:e9:16
>> >> [  343.777974] igb :01:00.2: eth0: PBA No: 106100-000
>> >> [  343.777977] igb :01:00.2: Using MSI-X interrupts. 4 rx queue(s), 4 
>> >> tx queue(s)
>> >> [  343.781905] igb :01:00.2 enp1s0f2: renamed from eth0
>> >> [  343.849424] IPv6: ADDRCONF(NETDEV_UP): enp1s0f2: link is not ready
>> >> [  344.139715] igb :01:00.3: added PHC on eth0
>> >> [  344.139718] igb :01:00.3: Intel(R) Gigabit Ethernet Network 
>> >> Connection
>> >> [  344.139722] igb :01:00.3: eth0: (PCIe:5.0Gb/s:Width x4) 
>> >> 00:12:c0:00:e9:17
>> >> [  344.139797] igb :01:00.3: eth0: PBA No: 106100-000
>> >> [  344.139800] igb :01:00.3: Using MSI-X interrupts. 4 rx queue(s), 4 
>> >> tx queue(s)
>> >> [  344.141054] igb :01:00.3 enp1s0f3: renamed from eth0
>> >> [  344.213393] IPv6: ADDRCONF(NETDEV_UP): enp1s0f3: link is not ready
>> >>
>> >> I googled for this error message and found the following patch:
>> >>
>> >> https://www.spinics.net/lists/netdev/msg515557.html
>> >>
>> >> I'm not sure if it addresses my problem, but it looks related. I haven't
>> &

Re: igb: device_attach: igb1 attach returned 6

[Kevin Bowling]

Well I took a look at the phy code and actually don't see that we
abosrbed those commits.  I'll try and audit e1000 vs linux in the next
24 hours but in the mean time if you could confirm linux-next or the
like fix or don't fix the phy we can at least know if this is the
right path or if we need to ask Intel networking for help.

Regards,

On Thu, Aug 23, 2018 at 12:19 AM, Kevin Bowling
 wrote:
> Great thanks for the info about it not working on Ubuntu too, that
> narrows down the problem space considerably.
>
> Can you try that patch on FreeBSD -CURRENT?  It should apply with
> minimal changes the phy code is shared by multiple OSes.  If it works
> we can commit.
>
> Regards,
>
> On Thu, Aug 23, 2018 at 12:16 AM, Roman Bogorodskiy
>  wrote:
>> I haven't used this hardware before, so I don't if it ever worked. I
>> only tried 11.2 and it doesn't work. I've also tried Ubuntu 18.04 and it
>> doesn't work as well. Ubuntu prints the following error:
>>
>> [  343.053031] igb: Intel(R) Gigabit Ethernet Network Driver - version 
>> 5.4.0-k
>> [  343.053032] igb: Copyright (c) 2007-2014 Intel Corporation.
>> [  343.415346] igb :01:00.0: added PHC on eth0
>> [  343.415349] igb :01:00.0: Intel(R) Gigabit Ethernet Network Connection
>> [  343.415352] igb :01:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 
>> 00:12:c0:00:e9:14
>> [  343.415427] igb :01:00.0: eth0: PBA No: 106100-000
>> [  343.415430] igb :01:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx 
>> queue(s)
>> [  343.416422] igb :01:00.0 enp1s0f0: renamed from eth0
>> [  343.416827] igb: probe of :01:00.1 failed with error -3
>> [  343.485242] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
>> [  343.777894] igb :01:00.2: added PHC on eth0
>> [  343.777897] igb :01:00.2: Intel(R) Gigabit Ethernet Network Connection
>> [  343.777900] igb :01:00.2: eth0: (PCIe:5.0Gb/s:Width x4) 
>> 00:12:c0:00:e9:16
>> [  343.777974] igb :01:00.2: eth0: PBA No: 106100-000
>> [  343.777977] igb :01:00.2: Using MSI-X interrupts. 4 rx queue(s), 4 tx 
>> queue(s)
>> [  343.781905] igb :01:00.2 enp1s0f2: renamed from eth0
>> [  343.849424] IPv6: ADDRCONF(NETDEV_UP): enp1s0f2: link is not ready
>> [  344.139715] igb :01:00.3: added PHC on eth0
>> [  344.139718] igb :01:00.3: Intel(R) Gigabit Ethernet Network Connection
>> [  344.139722] igb :01:00.3: eth0: (PCIe:5.0Gb/s:Width x4) 
>> 00:12:c0:00:e9:17
>> [  344.139797] igb :01:00.3: eth0: PBA No: 106100-000
>> [  344.139800] igb :01:00.3: Using MSI-X interrupts. 4 rx queue(s), 4 tx 
>> queue(s)
>> [  344.141054] igb :01:00.3 enp1s0f3: renamed from eth0
>> [  344.213393] IPv6: ADDRCONF(NETDEV_UP): enp1s0f3: link is not ready
>>
>> I googled for this error message and found the following patch:
>>
>> https://www.spinics.net/lists/netdev/msg515557.html
>>
>> I'm not sure if it addresses my problem, but it looks related. I haven't
>> tried that yet though because I don't have a spare drive to install
>> Linux and rebuild drivers there.
>>
>>   Kevin Bowling wrote:
>>
>>> The phy code hasn't changed in any material way that I can see in a
>>> long time.  So I wonder if it's timing or some initialization issue.
>>>
>>> e1000_get_hw_semaphore
>>> e1000_put_hw_semaphore
>>> e1000_read_phy_reg_i2c
>>>
>>> This looks a bit funny, I think the hw sem should be held over the i2c
>>> call.  Is there a chance you can get the same log from the working
>>> kernel?
>>>
>>> Regards,
>>> Kevin
>>>
>>> On Wed, Aug 22, 2018 at 2:08 AM, Roman Bogorodskiy
>>>  wrote:
>>> >   Kevin Bowling wrote:
>>> >
>>> >> Roman,
>>> >>
>>> >> Interesting.. can you set DBG to 1 sys/dev/e1000/e1000_osdep.h Line
>>> >> 109 so we can see closer what is failing to initialize?
>>> >
>>> > Hi Kevin,
>>> >
>>> > Here's a relevant bit of dmesg with debug enabled:
>>> >
>>> > igb1:  port 0xe040-0xe05f 
>>> > mem 0xf780-0xf79f,0xf7c08000-0xf7c0bfff irq 17 at device 0.1 on 
>>> > pci1
>>> > e1000_set_mac_type
>>> > igb1: attach_pre capping queues at 8
>>> > e1000_set_mac_type
>>> > e1000_init_mac_ops_generic
>>> > e1000_init_phy_ops_generic
>>> > e1000_init_nvm_ops_generic
>>> > e1000_init_function_pointers_82575
>>> > e1000_init_mac_params_82575
>>> > e

Re: igb: device_attach: igb1 attach returned 6

[Kevin Bowling]

Great thanks for the info about it not working on Ubuntu too, that
narrows down the problem space considerably.

Can you try that patch on FreeBSD -CURRENT?  It should apply with
minimal changes the phy code is shared by multiple OSes.  If it works
we can commit.

Regards,

On Thu, Aug 23, 2018 at 12:16 AM, Roman Bogorodskiy
 wrote:
> I haven't used this hardware before, so I don't if it ever worked. I
> only tried 11.2 and it doesn't work. I've also tried Ubuntu 18.04 and it
> doesn't work as well. Ubuntu prints the following error:
>
> [  343.053031] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
> [  343.053032] igb: Copyright (c) 2007-2014 Intel Corporation.
> [  343.415346] igb :01:00.0: added PHC on eth0
> [  343.415349] igb :01:00.0: Intel(R) Gigabit Ethernet Network Connection
> [  343.415352] igb :01:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 
> 00:12:c0:00:e9:14
> [  343.415427] igb :01:00.0: eth0: PBA No: 106100-000
> [  343.415430] igb :01:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx 
> queue(s)
> [  343.416422] igb :01:00.0 enp1s0f0: renamed from eth0
> [  343.416827] igb: probe of :01:00.1 failed with error -3
> [  343.485242] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
> [  343.777894] igb :01:00.2: added PHC on eth0
> [  343.777897] igb :01:00.2: Intel(R) Gigabit Ethernet Network Connection
> [  343.777900] igb :01:00.2: eth0: (PCIe:5.0Gb/s:Width x4) 
> 00:12:c0:00:e9:16
> [  343.777974] igb :01:00.2: eth0: PBA No: 106100-000
> [  343.777977] igb :01:00.2: Using MSI-X interrupts. 4 rx queue(s), 4 tx 
> queue(s)
> [  343.781905] igb :01:00.2 enp1s0f2: renamed from eth0
> [  343.849424] IPv6: ADDRCONF(NETDEV_UP): enp1s0f2: link is not ready
> [  344.139715] igb :01:00.3: added PHC on eth0
> [  344.139718] igb :01:00.3: Intel(R) Gigabit Ethernet Network Connection
> [  344.139722] igb :01:00.3: eth0: (PCIe:5.0Gb/s:Width x4) 
> 00:12:c0:00:e9:17
> [  344.139797] igb :01:00.3: eth0: PBA No: 106100-000
> [  344.139800] igb :01:00.3: Using MSI-X interrupts. 4 rx queue(s), 4 tx 
> queue(s)
> [  344.141054] igb :01:00.3 enp1s0f3: renamed from eth0
> [  344.213393] IPv6: ADDRCONF(NETDEV_UP): enp1s0f3: link is not ready
>
> I googled for this error message and found the following patch:
>
> https://www.spinics.net/lists/netdev/msg515557.html
>
> I'm not sure if it addresses my problem, but it looks related. I haven't
> tried that yet though because I don't have a spare drive to install
> Linux and rebuild drivers there.
>
>   Kevin Bowling wrote:
>
>> The phy code hasn't changed in any material way that I can see in a
>> long time.  So I wonder if it's timing or some initialization issue.
>>
>> e1000_get_hw_semaphore
>> e1000_put_hw_semaphore
>> e1000_read_phy_reg_i2c
>>
>> This looks a bit funny, I think the hw sem should be held over the i2c
>> call.  Is there a chance you can get the same log from the working
>> kernel?
>>
>> Regards,
>> Kevin
>>
>> On Wed, Aug 22, 2018 at 2:08 AM, Roman Bogorodskiy
>>  wrote:
>> >   Kevin Bowling wrote:
>> >
>> >> Roman,
>> >>
>> >> Interesting.. can you set DBG to 1 sys/dev/e1000/e1000_osdep.h Line
>> >> 109 so we can see closer what is failing to initialize?
>> >
>> > Hi Kevin,
>> >
>> > Here's a relevant bit of dmesg with debug enabled:
>> >
>> > igb1:  port 0xe040-0xe05f 
>> > mem 0xf780-0xf79f,0xf7c08000-0xf7c0bfff irq 17 at device 0.1 on 
>> > pci1
>> > e1000_set_mac_type
>> > igb1: attach_pre capping queues at 8
>> > e1000_set_mac_type
>> > e1000_init_mac_ops_generic
>> > e1000_init_phy_ops_generic
>> > e1000_init_nvm_ops_generic
>> > e1000_init_function_pointers_82575
>> > e1000_init_mac_params_82575
>> > e1000_read_sfp_data_byte
>> > e1000_read_sfp_data_byte
>> > e1000_init_nvm_params_82575
>> > e1000_init_phy_params_82575
>> > e1000_reset_mdicnfg_82580
>> > e1000_sgmii_uses_mdio_82575
>> > e1000_get_phy_id_82575
>> > e1000_sgmii_uses_mdio_82575
>> > e1000_read_phy_reg_sgmii_82575
>> > e1000_acquire_phy_82575
>> > e1000_acquire_swfw_sync
>> > e1000_get_hw_semaphore
>> > e1000_put_hw_semaphore
>> > e1000_read_phy_reg_i2c
>> > I2CCMD Error bit set
>> > e1000_release_phy_82575
>> > e1000_release_swfw_sync
>> > e1000_get_hw_semaphore
>> > e1000_put_hw_semaphore
>> > PHY address 1 was unreadable
>> > e1000_

Re: igb: device_attach: igb1 attach returned 6

[Kevin Bowling]

The phy code hasn't changed in any material way that I can see in a
long time.  So I wonder if it's timing or some initialization issue.

e1000_get_hw_semaphore
e1000_put_hw_semaphore
e1000_read_phy_reg_i2c

This looks a bit funny, I think the hw sem should be held over the i2c
call.  Is there a chance you can get the same log from the working
kernel?

Regards,
Kevin

On Wed, Aug 22, 2018 at 2:08 AM, Roman Bogorodskiy
 wrote:
>   Kevin Bowling wrote:
>
>> Roman,
>>
>> Interesting.. can you set DBG to 1 sys/dev/e1000/e1000_osdep.h Line
>> 109 so we can see closer what is failing to initialize?
>
> Hi Kevin,
>
> Here's a relevant bit of dmesg with debug enabled:
>
> igb1:  port 0xe040-0xe05f mem 
> 0xf780-0xf79f,0xf7c08000-0xf7c0bfff irq 17 at device 0.1 on pci1
> e1000_set_mac_type
> igb1: attach_pre capping queues at 8
> e1000_set_mac_type
> e1000_init_mac_ops_generic
> e1000_init_phy_ops_generic
> e1000_init_nvm_ops_generic
> e1000_init_function_pointers_82575
> e1000_init_mac_params_82575
> e1000_read_sfp_data_byte
> e1000_read_sfp_data_byte
> e1000_init_nvm_params_82575
> e1000_init_phy_params_82575
> e1000_reset_mdicnfg_82580
> e1000_sgmii_uses_mdio_82575
> e1000_get_phy_id_82575
> e1000_sgmii_uses_mdio_82575
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 1 was unreadable
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 2 was unreadable
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 3 was unreadable
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 4 was unreadable
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 5 was unreadable
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 6 was unreadable
> e1000_read_phy_reg_sgmii_82575
> e1000_acquire_phy_82575
> e1000_acquire_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> e1000_read_phy_reg_i2c
> I2CCMD Error bit set
> e1000_release_phy_82575
> e1000_release_swfw_sync
> e1000_get_hw_semaphore
> e1000_put_hw_semaphore
> PHY address 7 was unreadable
> PHY Initialization Error
> igb1: Setup of Shared code failed, error -2
> igb1: IFDI_ATTACH_PRE failed 6
> device_attach: igb1 attach returned 6
>
> I've also attached a complete dmesg.
>
>> Regards,
>> Kevin
>>
>> On Sun, Aug 19, 2018 at 7:21 AM, Roman Bogorodskiy  wrote:
>> > Hi,
>> >
>> > I have a 4-port Intel I350 network card. When I plug sfp rj45 module to
>> > one of the ports, devices fails to initialise, e.g. ifconfig(8) only
>> > shows three igb interfaces (expected to be four):
>> >
>> > igb0: flags=8802 metric 0 mtu 1500
>> > 
>> > options=e505bb
>> > ether 00:12:c0:00:e9:14
>> > media: Ethernet autoselect
>> > status: no carrier
>> > nd6 options=29
>> > igb1: flags=8802 metric 0 mtu 1500
>> > 
>> > options=e505bb
>> > ether 00:12:c0:00:e9:16
>> > media: Ethernet autoselect
>> > status: no carrier
>> > nd6 options=29
>> > igb2: flags=8802 metric 0 mtu

Re: igb: device_attach: igb1 attach returned 6

[Kevin Bowling]

Roman,

Interesting.. can you set DBG to 1 sys/dev/e1000/e1000_osdep.h Line
109 so we can see closer what is failing to initialize?

Regards,
Kevin

On Sun, Aug 19, 2018 at 7:21 AM, Roman Bogorodskiy  wrote:
> Hi,
>
> I have a 4-port Intel I350 network card. When I plug sfp rj45 module to
> one of the ports, devices fails to initialise, e.g. ifconfig(8) only
> shows three igb interfaces (expected to be four):
>
> igb0: flags=8802 metric 0 mtu 1500
> 
> options=e505bb
> ether 00:12:c0:00:e9:14
> media: Ethernet autoselect
> status: no carrier
> nd6 options=29
> igb1: flags=8802 metric 0 mtu 1500
> 
> options=e505bb
> ether 00:12:c0:00:e9:16
> media: Ethernet autoselect
> status: no carrier
> nd6 options=29
> igb2: flags=8802 metric 0 mtu 1500
> 
> options=e505bb
> ether 00:12:c0:00:e9:17
> media: Ethernet autoselect
> status: no carrier
> nd6 options=29
>
> dmesg has the following:
>
> igb0:  port 0xe060-0xe07f mem 
> 0xf7a0-0xf7bf,0xf7c0c000-0xf7c0 irq 16 at device 0.0 on pci1
> igb0: attach_pre capping queues at 8
> igb0: using 1024 tx descriptors and 1024 rx descriptors
> igb0: msix_init qsets capped at 8
> igb0: pxm cpus: 4 queue msgs: 9 admincnt: 1
> igb0: using 4 rx queues 4 tx queues
> igb0: Using MSIX interrupts with 5 vectors
> igb0: allocated for 4 tx_queues
> igb0: allocated for 4 rx_queues
> igb0: Ethernet address: 00:12:c0:00:e9:14
> igb0: netmap queues/slots: TX 4/1024, RX 4/1024
> igb1:  port 0xe040-0xe05f mem 
> 0xf780-0xf79f,0xf7c08000-0xf7c0bfff irq 17 at device 0.1 on pci1
> igb1: attach_pre capping queues at 8
> igb1: Setup of Shared code failed, error -2
> igb1: IFDI_ATTACH_PRE failed 6
> device_attach: igb1 attach returned 6
> igb1:  port 0xe020-0xe03f mem 
> 0xf760-0xf77f,0xf7c04000-0xf7c07fff irq 18 at device 0.2 on pci1
> igb1: attach_pre capping queues at 8
> igb1: using 1024 tx descriptors and 1024 rx descriptors
> igb1: msix_init qsets capped at 8
> igb1: pxm cpus: 4 queue msgs: 9 admincnt: 1
> igb1: using 4 rx queues 4 tx queues
> igb1: Using MSIX interrupts with 5 vectors
> igb1: allocated for 4 tx_queues
> igb1: allocated for 4 rx_queues
> igb1: Ethernet address: 00:12:c0:00:e9:16
> igb1: netmap queues/slots: TX 4/1024, RX 4/1024
> igb2:  port 0xe000-0xe01f mem 
> 0xf740-0xf75f,0xf7c0-0xf7c03fff irq 19 at device 0.3 on pci1
> igb2: attach_pre capping queues at 8
> igb2: using 1024 tx descriptors and 1024 rx descriptors
> igb2: msix_init qsets capped at 8
> igb2: pxm cpus: 4 queue msgs: 9 admincnt: 1
> igb2: using 4 rx queues 4 tx queues
> igb2: Using MSIX interrupts with 5 vectors
> igb2: allocated for 4 tx_queues
> igb2: allocated for 4 rx_queues
> igb2: Ethernet address: 00:12:c0:00:e9:17
> igb2: netmap queues/slots: TX 4/1024, RX 4/1024
>
> And in pciconf(8) it shows as:
>
> igb0@pci0:1:0:0:class=0x02 card=0x00101b6d chip=0x15228086 
> rev=0x01 hdr=0x00
> vendor = 'Intel Corporation'
> device = 'I350 Gigabit Fiber Network Connection'
> class  = network
> subclass   = ethernet
> none1@pci0:1:0:1:   class=0x02 card=0x00101b6d chip=0x15228086 
> rev=0x01 hdr=0x00
> vendor = 'Intel Corporation'
> device = 'I350 Gigabit Fiber Network Connection'
> class  = network
> subclass   = ethernet
> igb1@pci0:1:0:2:class=0x02 card=0x00101b6d chip=0x15228086 
> rev=0x01 hdr=0x00
> vendor = 'Intel Corporation'
> device = 'I350 Gigabit Fiber Network Connection'
> class  = network
> subclass   = ethernet
> igb2@pci0:1:0:3:class=0x02 card=0x00101b6d chip=0x15228086 
> rev=0x01 hdr=0x00
> vendor = 'Intel Corporation'
> device = 'I350 Gigabit Fiber Network Connection'
> class  = network
> subclass   = ethernet
>
> Any suggestions what could be wrong with that?
>
> The host is FreeBSD 12.0-ALPHA1 amd64 @ r337885. I don't know how/if it
> worked before, just plugged that in.
>
> Roman Bogorodskiy
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: [REGRESSION] Fresh CURRENT consume much more CPU on network traffic (vlans + routing + ipfw with NAT)

[Kevin Bowling]

This sounds like a known quirk of the Atom CPU architecture and iflib
-- can you try this patch https://reviews.freebsd.org/D16302 it should
help specifically on your hardware.

Regards,
Kevin

On Tue, Jul 17, 2018 at 6:03 AM, Lev Serebryakov  wrote:
> On 17.07.2018 10:54, Eugene Grosbein wrote:
>
>>>  I have "SOHO" router on Atom D2500 with FreeBSD CURRENT. It runs
>>> CURRENT for very long time (from 11-CURRENT times), and recently it
>>> start to consume much more CPU on same traffic — to the point when it
>>> becomes unresponsive in shell (via ssh, not local console).
>>>
>>>  I have rather complex ipfw ruleset, but this ruleset is the same for
>>> many years.
>>>
>>>  Revisions before r333989 worked normally, I never seen any problem with
>>> shell, no matter how much traffic is processed
>>>
>>>  Revision r334649 with same configuration, same firewall ruleset, etc.,
>>> becomes completely unresponsive under network load (pure transit traffic).
>>>
>>>  when system is unresponsive I see this in `top -SH`
>>>
>>> 100083 root -76 - 0K   272K -   1 291.8H  95.31% kernel{if_io_tqg_1}
>>> 100082 root -76 - 0K   272K -   0 297.7H  95.20% kernel{if_io_tqg_0}
>>>
>>>  And it is new to me.
>>
>> I'm sure you will get it solved more quick if you perform bisection of 
>> revision
>> even though it will take time.
>  I'll try latest version (seems here were a lot of commit to iflib after
> my revision) and after that try yo bisect.
>
> --
> // Lev Serebryakov
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Testing VF/PF code

[Kevin Bowling]

Harry,

I wasn’t aware of anyone desiring vf support for igb but I’ll take a look
at it for you if you can test -current with patches when I’m ready. My
motive is more to validate and refine the iflib functionality and this is a
good exercise.

On Wed, May 30, 2018 at 8:20 AM Harry Schmalzbauer 
wrote:

> Am 30.05.2018 um 16:45 schrieb Ryan Stone:
> > On Tue, May 29, 2018 at 12:58 PM Sean Bruno  wrote:
> >> Does anyone have a process for testing the VF drivers (ixgbe igb, etc)
> >> in FreeBSD without actually firing up linux to instantiate a VM or using
> >> EC2?
> > We have native support for creating VFs for ixl and ixgbe (and cxgbe).
> > For igb you're out of luck (but SR-IOV on igb is kind of a waste of
> > time anyway)
>
> I'd like to note that I'm strongly missing SR-IOV for if_igb(4) and I
> don't consider it as a waste of time, speaking of the time needed for
> the setup – not the time to make the code happen; that's nothing I can
> achive (not even estimate) so I won't try to judge about the sense of
> that time relation...
>
> 82576 is a wonderful piece of hardware (mostly true for i350 also) and
> I'm missing the ability to use VFs for jails and bhyve(8) likewise with
> these NICs.
> There are still many appliances that don't need 10GE rates and could
> easily cope with FastEthernet rates. For such appliances,
> security/design considerations have much more weight than throughput,
> which VFs would greatly support implementation simplicity/consistency.
> So igbv(4) is on top of my christmas wishlist ;-)
> Two or three of the 2-port cards and a LACP switch-stack (GbE) make a
> nice platform for a dozend VMs/jails – affordable by means of financial
> and electrical power budget likewise.
>
> No tech aspects/justification here, just my experience based opinion.
>
> Thanks,
>
> -harry
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Testing VF/PF code

[Kevin Bowling]

iovctl worked at one point with ixgbe. I don’t recall if there is any way
to attach the vf directly if you do that but you can use ppt driver to pass
them into bhyve.

On Tue, May 29, 2018 at 9:58 AM Sean Bruno  wrote:

> Does anyone have a process for testing the VF drivers (ixgbe igb, etc)
> in FreeBSD without actually firing up linux to instantiate a VM or using
> EC2?
>
> sean
>
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: svn commit: r333338 - in stable/11/sys: dev/bnxt kern net sys

[Kevin Bowling]

On Tue, May 8, 2018 at 2:43 AM, Harry Schmalzbauer  wrote:
> Bezüglich Kevin Bowling's Nachricht vom 08.05.2018 11:26 (localtime):
>> iflib in stable/11 only affects bnxt at this time.
>>
>> You should try out HEAD and let us know for the rest of your questions.
>
> Ic, sorry for the noise – should have read the commits before wasting
> others' time :-(
>
> Thanks for your help, but I can only briefly test hartwell and kawela
> (82574, Desktop 1gE and 82576, ET2 dual Server 1gE) and see how much
> queues they use, since I don't run -current on anything productive and
> spare time is even much more limited than spare machines ;-)
> Will find out who many queues i217 should provide as soon as possible
> and if it's more than one, I'll file a PR.
>
> But if the simple iflib/hw-support test with kawela+hartwell helps I'm
> happy to do.

At this point it would be helpful, we think e1000 is nearing pretty
good shape and I need to become familiar with any outstanding bugs.

> -harry
>
> P.S.: I guess I stumbled across your domain while searching for ADF and
> IML to get my 8590 running with a Adapter/A for Ethernet...
> Impressive important archive, thanks!

Worlds collide :D
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: svn commit: r333338 - in stable/11/sys: dev/bnxt kern net sys

[Kevin Bowling]

iflib in stable/11 only affects bnxt at this time.

You should try out HEAD and let us know for the rest of your questions.

Regards,
Kevin

On Tue, May 8, 2018 at 2:19 AM, Harry Schmalzbauer  wrote:
>  Bezüglich Stephen Hurd's Nachricht vom 07.05.2018 23:42 (localtime):
>> Author: shurd
>> Date: Mon May  7 21:42:22 2018
>> New Revision: 38
>> URL: https://svnweb.freebsd.org/changeset/base/38
>>
>> Log:
>>   Merge iflib changes to 11-STABLE
>>
>>   MFC r300147, r300153, r300154, r300215, r301563, r301567,
>>   r302372, r307560, r307562, r307563, r307568, r308792, r311039,
>>   r311837, r312755, r312903, r312905, r312924, r313248, r315217,
>>   r315245, r315288, r316278, r316281, r316502, r316596, r317756,
>>   r319917, r319921, r319984, r319989, r320059, r320609, r320611,
>>   r321253, r321629, r321630, r322337, r322338, r322823, r323077,
>>   r323825, r323876, r323879, r323887, r323941, r323942, r323943,
>>   r323944, r323954, r324038, r324318, r324937, r325166, r325167,
>>   r325168, r325201, r325241, r325245, r325487, r325494, r325901,
>>   r326033, r326369, r326370, r326432, r326577, r326578, r326702,
>>   r326706, r326775, r327013, r327017, r327052, r327072, r327098,
>>   r327242, r327244, r327247, r329651, r329742, r330289, r330715,
>>   r330721, r332419, r332422, r332729
>>
>>   Reviewed by:sbruno
>>   Approved by:re (delphij@)
>>   Sponsored by:   Limelight Networks
>>   Differential Revision:  https://reviews.freebsd.org/D15142
>
> Thanks to all for all your hard work pushing iflib.
>
> Unfortunately I haven't had time to check if EM_MULTIQUEUE suffers from
> a regression with iflib – previous em-attached NICs 2nd queue support.
> At least one i217 I'm using -current with, uses only one queue. But that
> was igb, not em, so I'm wondering if i217 ist limited to 1 queue – at
> least i210 supports 4 queues... hints welcome.
>
> But I have several oldish machines with hartwell (82574L, still running
> 10-stable), which showed much better delivery if multiple line speed
> clients were generating load and EM_MULTIQUEUE added support for 2
> queues (and ULE on SMP machine...).
>
> I'm also unsure about the current netmap state. When I last checked
> (some months ago), igb(->em)/netmap didn't work. If netmap support was
> restored, I missed that.
>
> Can you confirm that if EM_MULTIQUEUE and netmap are supposed to be
> still working with all the adapters which were supported by previous
> stable-11 (em+igb) code?
> Or do we need bold release notes that igb/em users must be aware of
> significant changes?
>
> Thanks,
>
> -harry
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Diagnosing terrible ixl performance

[Kevin Bowling]

These are all pretty much known (to intel and NF/LLNW) issues with the
ixl driver.  If you must run 11.1 you are best off just buying a
chelsio T580.

If you can run HEAD, or perhaps an eventual 11.3 MFC,
https://github.com/intel-wired-ethernet/freebsd/tree/ixl-iflib may fix
this but there are a few remaining issues before that hits HEAD.

Regards,

On Thu, Apr 19, 2018 at 9:03 PM, Garrett Wollman  wrote:
> I'm commissioning a new NFS server with an Intel dual-40G XL710
> interface, running 11.1.  I have a few other servers with this
> adapter, although not running 40G, and they work fine so long as you
> disable TSO.  This one ... not so much.  On the receive side, it gets
> about 600 Mbit/s with lots of retransmits.  On the *sending* side,
> though, it's not even able to sustain 10 Mbit/s -- but there's no
> evidence of retransmissions, it's just sending really really slowly.
> (Other machines with XL710 adapters are able to sustain full 10G.)
> There is no evidence of any errors on either the adapter or the switch
> it's connected to.
>
> So far, I've tried:
>
> - Using the latest Intel driver (no change)
> - Using the latest Intel firmware (breaks the adapter)
> - Disabling performance tweaks in loader.conf and sysctl.conf
> - Changing congestion-control algorithms
>
> Anyone have suggestions while I still have time to test this?  (My
> plan B is to fall back to an X520 card that I have in my spares kit,
> because I *know* those work great with no faffing about.)  Any
> relevant MIBs to inspect?
>
> The test I'm doing here is simple iperf over TCP, with MTU 9120.  It
> takes about 10 seconds for the sending side to complete, but buffers
> are severely constipated for 20 seconds after that (delaying all
> traffic, including ssh connections).
>
> I'm at the point of trying different switch ports just to eliminate
> that as a possibility.
>
> -GAWollman
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Calling nxge(4) users

[Kevin Bowling]

Agreed. Please cull the ‘ixgb’ driver for similar reasons

On Mon, Mar 26, 2018 at 4:05 PM Brooks Davis  wrote:

> On Sun, Mar 25, 2018 at 01:09:27PM -0700, Rodney W. Grimes wrote:
> > > The nxge(4) driver for the Neterion Xframe-I and Xframe-II 10GbE
> > > adapters has obvious bugs (by inspection) and it doesn't appear the
> > > company exists any more.  We'd like to see if there are any significant
> > > users of this device and plan to remove it from FreeBSD-12 if not.
> > >
> > > -- Brooks
> >
> > Neterion has been acquired by exar.
> >   https://www.eetimes.com/document.asp?doc_id=1173009
> >
> > They have the datasheet for the Xframe-I here:
> >   https://www.exar.com/ds/xframedatasheet.pdf
> >
> > The cards appear to be readily avaliable on ebay.
>
> Given that most of these cards appear to have been PCI-X and the product
> failed in the market, I'm not finding that compelling.  One could buy
> a Chelsio or Intel card for a similar price and have a supported driver.
>
> Unless someone has a significant deployment of these cards that will run
> past the EOL of FreeBSD 11 (2021) this driver is a net negative.
>
> -- Brooks
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: why not enable tcp_pmtud_blackhole_detect in default

[Kevin Bowling]

Cheng,

We run this in production at Limelight Networks (i.e toward a broad
spectrum of Internet hosts) and must to deal with some uncommon
network topology. There are currently some limitations as you point
out.

Like you say the signaling is not perfect and we do often clamp MSS
unnecessarily.  There is also no probing to see if we can expand the
MSS later.

I think those issues should be fixed up before it's enabled by default
and I don't know anyone working on it at the moment.

Regards,

On Wed, Mar 7, 2018 at 8:35 AM, Cui, Cheng  wrote:
> Dear all,
>
> Reading through the tcp blackhole detection code (support RFC 4821) in 
> FreeBSD including the recent bug fixes, I am wondering why is it still not 
> enabled in default? Given the fact that this implementation was a merge from 
> xnu, and the xnu has enabled it in default, do we have a plan to enable it in 
> default? Or is there any concern about the side-effect from it as performance 
> regression against some false positive blackhole event like a temporary link 
> flap, which is long enough to trigger a lower MSS but shorter than 6 RTO?
>
> https://opensource.apple.com/source/xnu/xnu-1456.1.26/bsd/netinet/tcp_timer.c.auto.html
>   << enabled in macOS 10.6
> https://reviews.freebsd.org/rS322967  << bug fixes
> https://reviews.freebsd.org/rS272720  << merge from xnu
>
> Thanks,
> --Cheng Cui
> NetApp Scale Out Networking
> https://netapp-meeting.webex.com/meet/chengc
>
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Remove 'ixgb' from HEAD

[Kevin Bowling]

The ixgb driver was used only briefly by one sub family of devices in
the first half of the 2000s.  It's not a power efficient card with a
massive on board xenpak.  The driver saw little specific (not
API/treewide stuff) maintenance post FreeBSD 5 in which the last real
intentional commit directly on the driver was by someone admitting
they didn't have a card but spotting a bunch of errors in the driver.
It's unlikely the driver is suitable for commercial use as it stands,
and I wouldn't trust it in a home server.

If someone actually uses ixgb and sends me one I will flatten it into
iflib ixgbe.  You can get one for $250 here :o)
https://www.ebay.com/itm/Intel-PRO-10GbE-SR-PCI-X-PXLA8591SR-Server-Adapter-/110895948350

Otherwise I propose it be removed in HEAD, which gives a generous
deorbiting period of September 30, 2021 in stable/11.

Regards,
Kevin
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Dell SAS controller / mpr?

[Kevin Bowling]

A 3008 is an IT-mode card.  I think the 3108 is an IR-mode only card.
I don't think dell does much to the firmwares, you can probably flash
them with Broadcom ones if needed.

On Fri, Oct 6, 2017 at 1:09 PM, Lee Brown  wrote:
> Hi All,
>
> I want to purchase a dell R330 system with a "SAS 12Gbps HBA External
> Controller", which I believe is this (pdf)
> 
> card, based on the LSI SAS 3008 controller, which seems to be supported by
> the mpr driver.
>
> Would anybody be so kind as to verify this looks right before I purchase a
> brick!
>
> Many thanks -- lee
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Impending NATM removal

[Kevin Bowling]

For consideration, stable/11 will be supported until September 30, 2021
which provides a pretty long window to transition.

On Fri, Apr 7, 2017 at 6:15 AM, Łukasz Wójcik 
wrote:

> Oh, and mbpool DMA syncing does not work at all and -- at least in patm
> driver -- causes various issues.
>
> This of course can be easily worked around if you choose to let ATM
> ecosystem stay. If my client agrees,
>
> I could also share a patch for patm to make it work with newer ProSUM
> hardware.
>
>
> -ŁW
>
>
>
> On 07-Apr-17 15:08, Łukasz Wójcik wrote:
>
>> Hi Brooks,
>>
>>
>> AFAIK Prosum still manufactures PROATM155M card that utilizes patm driver
>> (which is by the way a little bit outdated and does
>>
>> not support newer variants of ProSUM cards). I also have clients that
>> still use ATM and prosum cards and FreeBSD. My guess is that
>>
>> they're not the only ones. Just my 5cents. The other thing is that
>> ATM/NATM infrastructure in FreeBSD seems to never have been
>>
>> finished.
>>
>>
>> Thanks
>>
>> -ŁW
>>
>>
>> On 07-Apr-17 01:57, Brooks Davis wrote:
>>
>>> As previously threatened, I plan to remove NATM support next week.  This
>>> includes the drivers en(4), fatm(4), hatm(4), and patm(4).  None of
>>> these devices have been manufactured in the last 20 years so it is time
>>> to move on.
>>>
>>> The planned commit can be seen at https://reviews.freebsd.org/D9883
>>>
>>> -- Brooks
>>>
>>
>>
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: interface down, console output: igb3: TX(7) desc avail = 41, pidx = 308

[Kevin Bowling]

Sean Bruno committed a couple fixes to the watchdog code this week that
should at least allow for a usable TSO although the frequency of the
watchdog events is still cause for concern.  It seems some timeouts are
part of Intel's expectations during normal operations for several chipsets.

If you could share which exact NIC chipset you have I will check the
datasheets and see if we're missing anything.

Regards,

On Sat, Apr 1, 2017 at 6:41 PM, Ben Woods <woods...@gmail.com> wrote:

> On 27 March 2017 at 15:35, Kevin Bowling <kevin.bowl...@kev009.com> wrote:
>
>> Try turning TSO off.. i.e. ifconfig igb3 -tso or sysctl net.inet.tcp.tso=0
>>
>> The transition to iflib has exposed much jankiness in the Intel "shared
>> code" of the e1000 drivers.  In particular, the locking contracts may not
>> align with FreeBSD locking primitives.  I have boxes running the legacy
>> driver that are clearly reliant on the watchdog reset for steady state
>> which is unacceptable.  We are actively looking into this at LLNW, but
>> additional reports and help are appreciated.
>>
>
>
> Hi Kevin,
>
> Thanks for the reply. Sorry it took so long for me to get back to you, I
> first had to wait for the problem to be repeated.
>
> Indeed, running "ifconfig igb3 -tso" did fix the issue. It didn't seem to
> the first time, but I may have been too quick to judge. After re-enabling
> TSO and disabling it a second time, the problem stopped, and the interface
> immediately came up.
>
> Please let me know if there is anything I can do to try and help diagnose
> this. In the mean time, I have added net.inet.tcp.tso=0 to my
> /etc/sysctl.conf, so I don't think this will re-occur unless I remove it.
>
> Regards,
> Ben
>
> --
> From: Benjamin Woods
> woods...@gmail.com
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Error at the time of compiling netmap-fwd on -CURRENT

[Kevin Bowling]

Hi,

The DPRINTF macro in netmap-fwd is defined as #define *DPRINTF*(_fmt,
args...) if (verbose) dprintf(_fmt, ## args)

while  dprintf is defined as dprintf(int fd, const char * restrict format,
...); in stdio.h

You could try, simplistically, to change DPRINTF macro to #define
*DPRINTF*(_fmt,
args...) if (verbose) printf(_fmt, ## args)

to get going but there is likely more to the story.


Regards,


On Fri, Mar 24, 2017 at 4:19 PM, Caraballo-vega, Jordan A.
(GSFC-6062)[COMPUTER SCIENCE CORP]  wrote:

> Hello,
>
> I am trying to compile netmap-fwd from git
> (https://github.com/Netgate/netmap-fwd) on a Dell PE 530 with -CURRENT
> updated from today (03/24/17). With the dependencies installed ( # pkg
> install libucl libevent2 ) I run make and get the following error
> followed by 8 warnings:
>
> cc -O2 -fPIC -g -Wall -Wshadow -Wcast-qual -Wwrite-strings
> -Wredundant-decls -Wnested-externs -Winline -I/usr/local/include -c arp.c
> In file included from arp.c:51:
> ./util.h:33:5: error: conflicting types for 'dprintf'
> int dprintf(const char *, ...);
> ^
> /usr/include/stdio.h:364:6: note: previous declaration is here
> int  dprintf(int, const char * __restrict, ...) __printflike(2, 3);
>  ^
> arp.c:394:11: warning: incompatible pointer to integer conversion
> passing 'const char [44]' to parameter of type 'int'
>   [-Wint-conversion]
> DPRINTF("%s: discarding the packet, too short (%d).\n",
> ^~
> ./util.h:29:54: note: expanded from macro 'DPRINTF'
>
>
> Any idea or feedback would be greatly appreciated.
>
> Cheers,
>
> Jordan
>
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: interface down, console output: igb3: TX(7) desc avail = 41, pidx = 308

[Kevin Bowling]

Try turning TSO off.. i.e. ifconfig igb3 -tso or sysctl net.inet.tcp.tso=0

The transition to iflib has exposed much jankiness in the Intel "shared
code" of the e1000 drivers.  In particular, the locking contracts may not
align with FreeBSD locking primitives.  I have boxes running the legacy
driver that are clearly reliant on the watchdog reset for steady state
which is unacceptable.  We are actively looking into this at LLNW, but
additional reports and help are appreciated.

Regards,

On Fri, Mar 24, 2017 at 6:33 PM, Ben Woods  wrote:

> Morning!
>
> Since my recent update from FreeBSD12-current r313908 to r315466, I have
> noticed some strange behaviour with one of my network interfaces.
>
> The interface seems to work fine for a day or so, but on a number of
> occasions I have found it to be down, and constantly outputting the
> following message to the console every few seconds:
> igb3: TX(7) desc avail = 41, pidx = 308
> igb3: TX(7) desc avail = 41, pidx = 308
> igb3: TX(7) desc avail = 41, pidx = 308
> ...
>
> The problem is quickly worked around by issuing the following commands:
> # service netif stop igb3
> # service netif start igb3
>
> Details of this particular network interface card:
> $ pciconf -lv | grep igb3 -A4
> igb3@pci0:0:20:1: class=0x02 card=0x1f418086 chip=0x1f418086 rev=0x03
> hdr=0x00
> vendor = 'Intel Corporation'
> device = 'Ethernet Connection I354'
> class   = network
> subclass  = ethernet
>
> Any ideas what this could be, or how to investigate further?
>
> Regards,
> Ben
>
> --
> From: Benjamin Woods
> woods...@gmail.com
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Why is MSI-X support disabled on bce(4)?

[Kevin Bowling]

I think this would be a good candidate for iflib and can provide some
assistance from Matt and Sean if someone wants to try or we might get to it
eventually.  Check out man 9 iflibdd.  We had a lot of stability and
ordering issues adding multiqueue to FBSD em(4) similar to what Sephe did
in DFBSD's emx(4) that went away after using iflib queue management
routines.

On Sat, Mar 11, 2017 at 6:12 PM, Sepherosa Ziehau 
wrote:

> On Tue, Mar 7, 2017 at 11:10 PM, Kajetan Staszkiewicz
>  wrote:
> > Dnia poniedziałek, 6 marca 2017 16:06:03 CET Sepherosa Ziehau pisze:
> >> On Thu, Mar 2, 2017 at 10:02 PM, Kajetan Staszkiewicz
> >>
> >>  wrote:
> >> > To whom it might concern:
> >> >
> >> > Well, at least it does concern me. Why is support for multiple
> interrupts
> >> > and queues not enabled on bce(4)?
> >> >
> >> > Whole block of code is surrounded with #ifdef 0 ... #endif
> >> >
> >> > https://github.com/freebsd/freebsd/blob/master/sys/dev/
> bce/if_bce.c#L1108
> >>
> >> It involves much more work than the commented out MSI-X allocation, like
> >> this:
> >> https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/
> b42386ee03a4e688c8
> >> 64ba8d7094064c63d93dce?hp=be5708901d52be5534d5075eec706f5570b6a0f3
> >
> > That is sad news. Should I assume that porting this driver from
> DragonflyBSD
> > to FreeBSD would be impossible?
>
> I believe its doable, since before the MSI-X work, the code base is almost
> same.
>
> >
> >
> > --
> > | pozdrawiam / greetings | powered by Debian, FreeBSD and CentOS |
> > |  Kajetan Staszkiewicz  | jabber,email: vegeta()tuxpowered net  |
> > |Vegeta  | www: http://vegeta.tuxpowered.net |
> > `^---'
>
>
>
> --
> Tomorrow Will Never Die
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: about that DFBSD performance test

[Kevin Bowling]

Right off the bat, FreeBSD doesn't really understand NUMA in any sufficient
capacity.  Unfortunately at companies like the one I work at, we take that
to mean "OK buy a high bin CPU and only populate one socket" which serves
us well and may ultimately be the best value but does nothing to address
the reality that multi-socket NUMA systems are common place and FreeBSD
should run in the same league on them as other operating systems to be
taken seriously.  I'd be interested in seeing how the contenders look by
removing one of the CPUs as it might at least put a spotlight on that issue.

With respect to SO_REUSEPORT, we are investigating implementing this round
robin behavior right now.  We think librss is a better way to go for
content serving, but software will get written with Linux APIs in mind so I
think it'll be a good idea to have a distributing SO_REUSEPORT anyway for
software that does not understand librss natively.

I have no commercial need for kernel IP forwarding but Matt Macy is doing a
lot of driver work for me and we are at least trying to keep it in line
with the legacy drivers.  Since sephe's test was ixgbe, it'd be interesting
to test
https://github.com/mattmacy/networking/commits/HEAD_MERGE/iflib-ixgbe-current
which makes the FreeBSD ixgbe driver much more like cxgbe (same ring code,
improved coalescing, various prefetching insights for 10g+).  In talking
with Matt there's probably some low hanging fruit at the driver layer for
this workload around batching and distribution.  I do think the use of
polling(4) in the benchmark is a bit hacky, it should be possible to get
similar performance to Linux by doing batching and distribution as we are
fine tuning in the iflib project.  I'd be looking at netmap before
polling(4) in any new product development that does heavy forwarding.  But
there's probably more to do up stack in routing and I don't know of anyone
working on it with significant commercial backing.

Ultimately I have to say congrats to DFBSD.  They pulled that performance
off with a small team and very little to no commercial backing.  FreeBSD
has a lot of full time commercial contributors, and while this is generally
a good thing for longevity, I think workplace culture (even/especially at
fabled places, that just means you're super busy) defeats some of the lucid
insight and tenacity that ardent personal contributors are able to pull off
of their own accord.

Regards,

On Tue, Mar 7, 2017 at 9:00 PM, Eugene M. Zheganin 
wrote:

> Hi.
>
> Some have probably seen this already - http://lists.dragonflybsd.org/
> pipermail/users/2017-March/313254.html
>
> So, could anyone explain why FreeBSD was owned that much. Test is split
> into two parts, one is nginx part, and the other is the IPv4 forwarding
> part. I understand that nginx ownage was due to SO_REUSEPORT feature, which
> we do formally have, but in DFBSD and Linux it does provide a kernel socket
> multiplexor, which eliminates locking, and ours does not. I have only found
> traces of discussion that DFBSD implementation is too hackish. Well,
> hackish or not, but it's 4 times faster, as it turns out. The IPv4
> forwarding loss is pure defeat though.
>
> Please not that although they use HEAD it these tests, they also mention
> that this is the GENERIC-NODEBUG kernel which means this isn't related to
> the WITNESS stuff.
>
> Please also don't consider this trolling, I'm a big FreeBSD fan through
> the years, so I'm asking because I'm kind of concerned.
>
> Eugene.
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Intel Dual Band Wireless AC 8260

[Kevin Bowling]

It used to work pretty well, but HEAD regressed heavily.  My card crashes
8260 non-stop.

I'm trying to resync everything to DFBSD, I will post a review if
successful.

On Mon, Feb 20, 2017 at 4:50 PM, Andreas Nilsson  wrote:

> Hello,
>
> I have a laptop with an Intel Dual Band Wireless AC 8260 card (
> iwm0@pci0:4:0:0:
>class=0x028000 card=0x11308086 chip=0x24f38086 rev=0x3a hdr=0x00 )
> in my thinkpad x1 yoga.
>
> I'm running 12-CURRENT, and wireless is sort of working, although very
> slowly: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a where I would
> expect more. My old trusted access point is a wndr3700 which supports 11n (
> worked fine with my macbook, regularly got 20Mb/s.
>
> I bought a new access point to get that 11ac goodness, but I can barley get
> my new laptop to find the anything on the 5GHz band.
>
> So my questions:
> -Does FreeBSD actually support 11ac yet?
> -Shouldn't my ac8260 support 11n as well?
> -Should I get different scan results depending on if have my old or new ap
> connected?
> -How come my t510 thinkpad (different os btw) can see my 5GHz net on my new
> ap but my yoga does not?
>
> Best regards
> Andreas
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Disappointing packets-per-second performance results on a Dell, PE R530

[Kevin Bowling]

I would also suggest increasing the holdoff timer for the Chelsio and see
what happens:

sysctl dev.t5nex.0.holdoff_timer_idx_10G=3
sysctl dev.t5nex.0.holdoff_timer_idx_10G=4
sysctl dev.t5nex.0.holdoff_timer_idx_10G=5

On Tue, Jan 31, 2017 at 11:10 AM, Slawa Olhovchenkov  wrote:

> On Tue, Jan 31, 2017 at 01:53:15PM -0400, Jordan Caraballo wrote:
>
> > This are the most recent stats. No advances so far. The system has
> > -Current right now.
>
> This is like you have only one flow: only CPU 26 and 27 loaded.
>
> Use different benchmark, create multiple IP flow (w/ different
> source/destination IP pairs)
>
> > top -PHS
> > last pid: 18557; load averages: 2.58, 1.90, 0.95 up 4+00:39:54 18:30:46
> > 231 processes: 40 running, 126 sleeping, 65 waiting
> > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 4: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 6: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
> > CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 8: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 10: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 12: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 16: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 18: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 20: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 22: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 24: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 26: 0.0% user, 0.0% nice, 0.0% system, 59.6% interrupt, 40.4% idle
> > CPU 27: 0.0% user, 0.0% nice, 0.0% system, 96.3% interrupt, 3.7% idle
> > CPU 28: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 31: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 32: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 33: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> > CPU 34: 0.0% user, 0.0% nice, 0.0% system, 100% interrupt, 0.0% idle
> > CPU 35: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

RFC: ethctl

[Kevin Bowling]

Greetings,

I'm casting a wide net in cc, try to keep the noise minimal but we need
some input from a variety of HW vendors.

I have heard from several vendors the need for a NIC configuration tool.
 Chelsio ships a cxgb/cxgbetool in FreeBSD as one example.  There is
precedence for some nod toward a basic unified tool in Linux ethtool.

>From your perspective,
1) What are the common requirements?
2) What are specialized requirements? For instance as a full TCP offload
card Chelsio needs things others wont
3) What should it _not_ do?  Several of you have experience doing Ethernet
driver dev on many platforms so we should attempt to avoid repeating past
design mistakes.

I expect we can achieve some level of inversion so the device specific code
can live close to the driver and plug into the ethctl framework.  It should
be general enough to add completely new top level commands, so vendors can
implement HW specific features.  On the other hand, we should attempt to
hook into common core for features every NIC provides, with a focus on
iflib.

I will fund Matt Macy to do the overall design and implementation.

Regards,
Kevin Bowling, on behalf of Limelight Networks for this effort
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: 10Gb on budged with fiber — what should IP choose?

[Kevin Bowling]

You should be able to find T420-CR or T420-BT for half that.  The primary
difference is that cxgbe is still a supported driver, and it's been high
quality from the start unlike a lot of other vendors.  You are engaging in
archeology with the other cards you mentioned, and may spend way more value
in time than you would in the purchase cost of two NICs :)

On Sun, Jan 15, 2017 at 6:47 AM, Lev Serebryakov  wrote:

> Hello Kevin,
>
> Sunday, January 15, 2017, 1:03:16 PM, you wrote:
>
> > Intel is overpriced and still riding their Ethernet reputation from well
> > over a decade ago with little to show for it.  I would recommend cxgbe
> > anything, so Chelsio T420 and up.  The T520-SO-CR is within reason for
> > demanding home office/small office users to purchase new and I've
> deployed
> > over a thousand of these at work with flawless performance.  T420 are a
> bit
> > cheaper on i.e. US ebay if that is viable to you but not by too much.  I
> > bought a T420-BT that way for home use and have no complaints.
>   Looks like our views of term "budget" is very different :). If I could
> spend
>  $200+/card then it would be noo-rainer.
>
> > You can use DAC twinax cables to connect 10G SFP+ ports together.  This
> > will save several watts of power and heat on each side vs 10G Base-T.
>  Twinax is not what I could fit in my old house's (not datacenter!) cable
> ducts
>  :( And new cable ducts means a LOT of dusty work (reinforced concrete
>  walls) & new wallpapers & stuff.
>
>
> --
> Best regards,
>  Levmailto:l...@freebsd.org
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: 10Gb on budged with fiber — what should IP choose?

[Kevin Bowling]

Intel is overpriced and still riding their Ethernet reputation from well
over a decade ago with little to show for it.  I would recommend cxgbe
anything, so Chelsio T420 and up.  The T520-SO-CR is within reason for
demanding home office/small office users to purchase new and I've deployed
over a thousand of these at work with flawless performance.  T420 are a bit
cheaper on i.e. US ebay if that is viable to you but not by too much.  I
bought a T420-BT that way for home use and have no complaints.  For non-TOE
use at 10g, the difference between T4 and T5 doesn't matter too much, pcie
2.0 vs 3.0.

You can use DAC twinax cables to connect 10G SFP+ ports together.  This
will save several watts of power and heat on each side vs 10G Base-T.
Sometimes DACs have interop problems, usually with switches.  Optics are
more forgiving, although again the switch side may have extra hoops like
approved vendors.

As far as a switch, I got a good deal on the beta UniFi version of
https://www.ubnt.com/edgemax/edgeswitch-16-xg/ for my home network.  Now
that it is full price, I would recommend the EdgeMax version unless you
have other UniFi gear.  They are inexpensive as far as the physical
connectivity versus the rest of the market but not quite cheap yet.  These
aren't exactly pro grade, the interop seems pretty poor in general, but
that's the price you pay for the price you didn't pay :)

Regards,

On Sun, Jan 15, 2017 at 2:32 AM, Lev Serebryakov  wrote:

> Hello Freebsd-net,
>
>   I want to attach my DIY-NAS (FreeBSD-based) to my desktop (Windows-based)
>  with 10Gb link. I could not afford 10Gb switch for sure. So, it will be
>  point-to-point connection only for my desktop and not other computers in
> my
>  house.
>
>   Also, I could not use my current twisted pair wiring, as I need
> connection
>  to router for my desktop too :) Cable ducts ion my house is overcrouded,
>  so, I think only fiber is way to go, there is no chance to add CAT6 cable
>  between my room and storage room with server, and copper SFP-to-SFP
>  patchcord is out of question!
>
>   So, I need cheap PCIe cards with SFP+ slots which is not very picky about
>  modules (I'm on budget!), with good FreeBSD support and drivers for
>  Win'10 (which is another problem with very old cards).
>
>   What should I choose? There are A LOT of old 10Gb cards on eBay. Should I
>  choose Mellanox ConnectiX-2? Chelsio-T3? Something else? Intel X520 looks
>  as safe variant, but it is much more expensive and, as far as I know, it
>  supports only expensive branded SFP+ modules, too.
>
>
> P.S. You could never stop upgrading your home NAS.
>
> --
> Best regards,
>  Lev  mailto:l...@serebryakov.spb.ru
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

ixgbe PF and ixv driver

[Kevin Bowling]

Hi,

I've been trying to test SR-IOV on -CURRENT and 11.0-RELEASE.

Physical NIC is an X552.  The ixgbe PF has to be limited to 1-4 queues in
order to create VFs or you will get ENOSPC from iovctl.

Either letting ixv try to attach on the host, or creating a passthrough and
allowing pfsense 2.4.0 beta to try to attach yields this failure to attach:
m 0xc0104000-0xc0107fff,0xc0108000-0xc010bfff at device 7.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: ixgbe_reset_hw() failed with error -10
device_attach: ixv0 attach returned 5

Another issue, the PF is unable to pass traffic after creating a VF.
'iovctl -D -d ix0; service netif restart; service routing restart' will
restore connectivity.

Regards,
Kevin
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: FreeBSD 10 - ixgbe packet drop

[Kevin Bowling]

On 6/4/2014 11:46 PM, Özkan KIRIK wrote:

Hi

I'm using FreeBSD 10.
My ix0 is connected to my backbone switch.
Traffic is about 90Mbit/s.
But after 3 minutes it stops working.
ifconfig ix0 down ; ifconfig ix0 up solves problem temprorarily.

It's strange that, dev.ix.0.dropped is 0 but, netstat's Idrop counter is
growing.
How can i debug this situation?




dev.ix.0.mac_stats.local_faults: 33864
dev.ix.0.mac_stats.remote_faults: 18168


These faults look similar to a problem I have with the X520 using copper 
twinax cables.


What is your physical topology including cabling type?

___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

Re: Multihomed system with jails routing issues

[Kevin Bowling]

On 4/6/2014 6:36 AM, Julian Elischer wrote:

On 4/6/14, 5:04 PM, Chris Smith wrote:

On 06/04/14 04:20, Julian Elischer wrote:
Hey Julian,

Thanks for that. I did come across it but all of the documentation I
found indicated that it was experimental.

After a day or so messing around with VIMAGE/vnet and their various
gotchas and interactions with jails on FreeBSD 10, I have something
working that I'm happy with.


as long as you steer clear of pf and do only 'vanilla' stuff, you should
be ok.
let us know what you think and I'd like to see your notes published, if
not officially then at least put here so that others can find it in the
archives.


There have been long standing memory leaks in stopping VNET jails.  For 
instance kern/164763.


Is there anyone looking into this?  Is there any will to enable VNET by 
default in -CURRENT?


Regards,
Kevin

___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

Re: VNET, if_bridge, if_epair, vlans and bridged phy?

[Kevin Bowling]

On 3/16/2014 8:04 PM, Kevin Bowling wrote:

I'm trying a somewhat elaborate VNET jails setup and for the most part
it's working.  I'm using if_epairs, one side that gets passed into the
jail, and the other side that attaches to an if_bridge.  The if_bridge
has a member on a vlan interface.  So far so good.

cloned_interfaces=bridge0 bridge1 bridge2 vlan0 vlan1
ifconfig_ix0=inet pub ip netmask 255.255.255.240 up
ifconfig_vlan0=vlan 1010 vlandev ix0
ifconfig_vlan1=vlan 1011 vlandev ix0
ifconfig_bridge1=inet 10.10.10.55/24 addm vlan0 description vlan0
ifconfig_bridge2=inet 10.10.11.55/24 addm vlan1 description vlan1

The above works fine, the VNET jails are able to access the outside
world and vis versa (NAT happens on a dedicated router, not this host).

Now, if I instead do something like this to add the public IP to a bridge:

ifconfig_ix0=up
ifconfig_vlan0=vlan 1010 vlandev ix0
ifconfig_vlan1=vlan 1011 vlandev ix0
ifconfig_bridge0=inet pub ip netmask 255.255.255.240 addm ix0
description ix0
ifconfig_bridge1=inet 10.10.10.55/24 addm vlan0 description vlan0
ifconfig_bridge2=inet 10.10.11.55/24 addm vlan1 description vlan1

A VNET jail on bridge0 in the public IP space works fine, but bridge1
and bridge2 are no longer accessible from the outside, including the
host interface like 10.10.10.55.

Any ideas on what could be going wrong?  Is there a way to use an
untagged interface like this in addition to the tagged ones?

Regards,
Kevin


I'm able to work around this by setting the native VLAN on the switch to 
a bogus value and using another tagged interface for the public IP (now 
nothing uses untagged interface).


I'm guessing it might be rstp/mstp related since STP does not happen on 
the VLAN interfaces, but it does on the native port when added to a 
bridge.  When they're all VLANs, I don't think if_bridge will send any 
BPDUs to the switch.


Regards,
Kevin


___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

VNET, if_bridge, if_epair, vlans and bridged phy?

[Kevin Bowling]

I'm trying a somewhat elaborate VNET jails setup and for the most part 
it's working.  I'm using if_epairs, one side that gets passed into the 
jail, and the other side that attaches to an if_bridge.  The if_bridge 
has a member on a vlan interface.  So far so good.


cloned_interfaces=bridge0 bridge1 bridge2 vlan0 vlan1
ifconfig_ix0=inet pub ip netmask 255.255.255.240 up
ifconfig_vlan0=vlan 1010 vlandev ix0
ifconfig_vlan1=vlan 1011 vlandev ix0
ifconfig_bridge1=inet 10.10.10.55/24 addm vlan0 description vlan0
ifconfig_bridge2=inet 10.10.11.55/24 addm vlan1 description vlan1

The above works fine, the VNET jails are able to access the outside 
world and vis versa (NAT happens on a dedicated router, not this host).


Now, if I instead do something like this to add the public IP to a bridge:

ifconfig_ix0=up
ifconfig_vlan0=vlan 1010 vlandev ix0
ifconfig_vlan1=vlan 1011 vlandev ix0
ifconfig_bridge0=inet pub ip netmask 255.255.255.240 addm ix0 
description ix0

ifconfig_bridge1=inet 10.10.10.55/24 addm vlan0 description vlan0
ifconfig_bridge2=inet 10.10.11.55/24 addm vlan1 description vlan1

A VNET jail on bridge0 in the public IP space works fine, but bridge1 
and bridge2 are no longer accessible from the outside, including the 
host interface like 10.10.10.55.


Any ideas on what could be going wrong?  Is there a way to use an 
untagged interface like this in addition to the tagged ones?


Regards,
Kevin

___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org