BGP Router Hardware Suggestions
We are about to discover the joys of upstream BGP routing :-P The current plan is to use a pair of OpenBSD+bgpd hosts as the routers. Each host will require 4x10gig ports (SFP+). One of those links (to AWS) will be close to saturated, along with the downlink to our switches. The other two will only need to carry ~1Gb/s of traffic. We are pretty much a Supermicro shop, and I'm wondering if anyone out there is running a similar setup on SM hardware. My main concern is finding NICs that will let us squeeze every last drop of bandwidth on the 10gig links. I did run some brief ttcp tests on a pair of SM 1Us (don't have the model number handy, maybe 5018-FTN4s?) with add-in Intel cards (550s?) and was able to get 700 MBytes/s of throughput. This would have been circa the 6.7 or 6.8 releases. I'm hoping to get >70% of the theoretical bandwidth out of the new hardware, and my gut says it's the NIC that's constraining us. So, I'd be interested in hearing from anyone running a similar setup, or who has benchmarked any of the current crop of 10gig NICs and has good/bad things to say about specific models. Thanks, --lyndon
Re: OpenBSD 7.3 ixl SIOCSIFMEDIA: Operation not supported
On 6/29/23 16:34, Rachel Roch wrote: Looking at some of the release notes staying on newer firmware would obviously be more preferable, but I'll keep downgrading as a last-resort if the other side of the link can't turn on autoneg. If you end up downgrading, I would be interested in knowing the version you settle on. I may end up downgrading myself because I would like to set the media value and options if I could.
Re: OpenBSD 7.3 ixl SIOCSIFMEDIA: Operation not supported
29 Jun 2023, 15:58 by z...@philomathiclife.com: > Just checked again, and my memory did not betray me: I am unable to set > the media let alone media options. It is likely the API being too new. > ixl(4) does state that one should flash their card with "the most recent > (stable) firmware", but it appears this firmware is too new. You can > try progressively downgrading your firmware to the newest version that > does have a working API: > > https://www.intel.com/content/www/us/en/download/683939/nvm-downgrade-packages-for-intel-ethernet-adapters-700-series.html > Thanks Zack. Looking at some of the release notes staying on newer firmware would obviously be more preferable, but I'll keep downgrading as a last-resort if the other side of the link can't turn on autoneg.
Re: OpenBSD 7.3 ixl SIOCSIFMEDIA: Operation not supported
Just checked again, and my memory did not betray me: I am unable to set the media let alone media options. It is likely the API being too new. ixl(4) does state that one should flash their card with "the most recent (stable) firmware", but it appears this firmware is too new. You can try progressively downgrading your firmware to the newest version that does have a working API: https://www.intel.com/content/www/us/en/download/683939/nvm-downgrade-packages-for-intel-ethernet-adapters-700-series.html
Re: relayd: pfe_route: failed to add gateway 22 Invalid argument
On Thu, Jun 29, 2023 at 09:34:10AM +0200, Jörg Streckfuß wrote: > Hi Claudio, > > Am 29.06.23 um 09:01 schrieb Claudio Jeker: > > On Thu, Jun 29, 2023 at 08:53:05AM +0200, Jörg Streckfuß wrote: > > > > > > Hi list, > > > > > > here is a small addition. Adding and deleting the route to and from > > > routing > > > table on the command line works as expected: > > > > > > fw1 # route add 2001:::::4/128 2001:::::4 -label > > > geo_service > > > add host 2001:::::4/128: gateway 2001:::::4 > > > > > > fw# route -n show -inet6 | grep 2001:::::4 > > > 2001:::::4 52:01:8d:e4:fd:63 UHLch > > > 123015 - 3 vlan18 > > > 2001:::::4 2001:::::4 UGHS > > > 00 - 8 vlan18 > > > > > > fw1 # route del 2001:::::4/128 2001:::::4 -label > > > geo_service > > > del host 2001:::::4/128: gateway 2001:::::4 > > > > > > fw1# route -n show -inet6 | grep 2001:::::4 > > > > > > 2001:638:dfce:3000::4 52:01:8d:e4:fd:63 UHLc > > > 0 > > > 23015 - 3 vlan18 > > > > > > > > > Why can't relayd add the route to the table and what does the following > > > log > > > concretely mean: > > > > > > > > > pfe_route: failed to add gateway 2001:638:dfce:3000::4: 22 Invalid > > > argument > > > > > > > > > > Run route -n monitor will give you more insights at what is sent to the > > kernel. At least unless the route message is so mangled that the kernel > > fails to parse it. > > This is interesting. I ran route -n monitor and run relayd but it says > nothing. No output at all. Ugh, relayd pfe_route is utterly broken for IPv6. That code never worked, the encoded message is not properly aligned because struct sockaddr_in6 does not align to a long word boundary. This function requires a rewrite. -- :wq Claudio
Re: relayd: pfe_route: failed to add gateway 22 Invalid argument
Hi Claudio, Am 29.06.23 um 09:01 schrieb Claudio Jeker: On Thu, Jun 29, 2023 at 08:53:05AM +0200, Jörg Streckfuß wrote: Hi list, here is a small addition. Adding and deleting the route to and from routing table on the command line works as expected: fw1 # route add 2001:::::4/128 2001:::::4 -label geo_service add host 2001:::::4/128: gateway 2001:::::4 fw# route -n show -inet6 | grep 2001:::::4 2001:::::4 52:01:8d:e4:fd:63 UHLch 123015 - 3 vlan18 2001:::::4 2001:::::4 UGHS 00 - 8 vlan18 fw1 # route del 2001:::::4/128 2001:::::4 -label geo_service del host 2001:::::4/128: gateway 2001:::::4 fw1# route -n show -inet6 | grep 2001:::::4 2001:638:dfce:3000::4 52:01:8d:e4:fd:63 UHLc 0 23015 - 3 vlan18 Why can't relayd add the route to the table and what does the following log concretely mean: pfe_route: failed to add gateway 2001:638:dfce:3000::4: 22 Invalid argument Run route -n monitor will give you more insights at what is sent to the kernel. At least unless the route message is so mangled that the kernel fails to parse it. This is interesting. I ran route -n monitor and run relayd but it says nothing. No output at all. Also all IPs does not help to understand what is going on. I will rebuild my setup to 2001:db adresses so I can post the full ipv6 address. Sorry this is because of a company policy. Why do you add /128 route with the same IP as the gateway? That just makes no sense. The idea is to check the availability of a service and announce the ip address to a bgp peer only if the service is up. I know it looks weird but for ipv4 it works. My naive assumption was to simply adopt it to IPv6. Is there a more elegant way to do this? Regards Joerg
Re: OpenBSD 7.3 ixl SIOCSIFMEDIA: Operation not supported
29 Jun 2023, 07:26 by stu.li...@spacehopper.org:
> On 2023-06-28, Rachel Roch wrote:
>
>> Sadly the other side of the link is beyond my control, different company.
>> I'll certainly ask if they can turn on autonegotiation.
>>
>
> autonegotiation isn't really a thing with fibre.
>
> "media autoselect" (i.e. the default) on fibre nics on OpenBSD isn't the
> same thing as autonegotiation on copper. if you already get link, there
> is probably no more to be done on your side.
>
Except the problem here is I'm not getting link ("status: no carrier").
Physical cabling is confirmed and light presence has been confirmed
independently on both sides. Which is why my mind was wandering towards
autonegotiation.
>
> not being able to force media might prevent talking at 1G with a 10G
> SFP+, but that's usually easiest solved by using a 1G SFP in the SFP+
> slot instead.
>
> accessing stats from DOM (ifconfig ixl sff) probably requires chasing
> API changes in the NIC firmware - those are running much newer firmware
> than my working ixl(4) cards:
>
> ixl3 at pci1 dev 0 function 3 "Intel X710 SFP+" rev 0x02: port 1, FW
> 6.80.48603 API 1.7, msix, 2 queues, address f8:f2:1e:...
>
I see, thanks.
Re: relayd: pfe_route: failed to add gateway 22 Invalid argument
On Thu, Jun 29, 2023 at 08:53:05AM +0200, Jörg Streckfuß wrote: > > Hi list, > > here is a small addition. Adding and deleting the route to and from routing > table on the command line works as expected: > > fw1 # route add 2001:::::4/128 2001:::::4 -label > geo_service > add host 2001:::::4/128: gateway 2001:::::4 > > fw# route -n show -inet6 | grep 2001:::::4 > 2001:::::4 52:01:8d:e4:fd:63 UHLch > 123015 - 3 vlan18 > 2001:::::4 2001:::::4 UGHS > 00 - 8 vlan18 > > fw1 # route del 2001:::::4/128 2001:::::4 -label > geo_service > del host 2001:::::4/128: gateway 2001:::::4 > > fw1# route -n show -inet6 | grep 2001:::::4 > > 2001:638:dfce:3000::4 52:01:8d:e4:fd:63 UHLc 0 > 23015 - 3 vlan18 > > > Why can't relayd add the route to the table and what does the following log > concretely mean: > > > pfe_route: failed to add gateway 2001:638:dfce:3000::4: 22 Invalid argument > > Run route -n monitor will give you more insights at what is sent to the kernel. At least unless the route message is so mangled that the kernel fails to parse it. Also all IPs does not help to understand what is going on. Why do you add /128 route with the same IP as the gateway? That just makes no sense. -- :wq Claudio
Re: relayd: pfe_route: failed to add gateway 22 Invalid argument
Hi list,
here is a small addition. Adding and deleting the route to and from
routing table on the command line works as expected:
fw1 # route add 2001:::::4/128 2001:::::4 -label
geo_service
add host 2001:::::4/128: gateway 2001:::::4
fw# route -n show -inet6 | grep 2001:::::4
2001:::::4 52:01:8d:e4:fd:63 UHLch
123015 - 3 vlan18
2001:::::4 2001:::::4 UGHS
00 - 8 vlan18
fw1 # route del 2001:::::4/128 2001:::::4 -label
geo_service
del host 2001:::::4/128: gateway 2001:::::4
fw1# route -n show -inet6 | grep 2001:::::4
2001:638:dfce:3000::4 52:01:8d:e4:fd:63 UHLc
023015 - 3 vlan18
Why can't relayd add the route to the table and what does the following
log concretely mean:
pfe_route: failed to add gateway 2001:638:dfce:3000::4: 22 Invalid argument
Am 28.06.23 um 16:57 schrieb Joerg Streckfuss:
Hello,
I'm trying to use the relayd router function to add host routes to the
routing table with a route label for further processing by bgpd. The
host ist directly connected to the firewall.
relayd.conf:
table { 2001:::::4 }
router "service_v6" {
route 2001:::::4/128
forward to port 80 check tcp
rtlabel geo_service
}
fw1# relayd -vvvd
startup
socket_rlimit: max open files 1024
pfe: filter init done
socket_rlimit: max open files 1024
socket_rlimit: max open files 1024
socket_rlimit: max open files 1024
parent_tls_ticket_rekey: rekeying tickets
hce_notify_done: 2001:::::4 (tcp connect ok)
host 2001:::::4, check tcp (0ms,tcp connect ok), state
unknown -> up, availability 100.00%
pfe_dispatch_hce: state 1 for host 1 2001:::::4
sync_routes: router service_v6 route 2001:::::4/128 gateway
2001:::::4 up priority 0
hce_notify_done: 2001:::::4 (tcp connect ok)
pfe_route: failed to add gateway 2001:::::4: 22 Invalid
argument
hce_notify_done: 2001:::::4 (tcp connect ok)
hce_notify_done: 2001:::::4 (tcp connect ok)
hce_notify_done: 2001:::::4 (tcp connect ok)
The route with the route label never pops up in the routing table. With
IPv4 addresses the setup works as expected.
Any suggestions?
Re: OpenBSD 7.3 ixl SIOCSIFMEDIA: Operation not supported
On 2023-06-28, Rachel Roch wrote: > Sadly the other side of the link is beyond my control, different company. > I'll certainly ask if they can turn on autonegotiation. autonegotiation isn't really a thing with fibre. "media autoselect" (i.e. the default) on fibre nics on OpenBSD isn't the same thing as autonegotiation on copper. if you already get link, there is probably no more to be done on your side. not being able to force media might prevent talking at 1G with a 10G SFP+, but that's usually easiest solved by using a 1G SFP in the SFP+ slot instead. accessing stats from DOM (ifconfig ixl sff) probably requires chasing API changes in the NIC firmware - those are running much newer firmware than my working ixl(4) cards: ixl3 at pci1 dev 0 function 3 "Intel X710 SFP+" rev 0x02: port 1, FW 6.80.48603 API 1.7, msix, 2 queues, address f8:f2:1e:...
