Hi Diederik,
I will try. Meanwhile I was troubleshooting this issue for some time and
I notice a change in FRRouting between 9.1 and 10.0.
Before 10.0 FRRouting was installing the routes in kernel using the
destination interface of the route. Starting from 10.0 FRRouting is
installing all routes towards the VRF interface.
Here is my bug reported on FRRouting:
https://github.com/FRRouting/frr/issues/15909
Example:
Working scenario with FRR 9.0.2 and 9.1:
|root@FRR01:/opt/Kitts/frr/9.0.2# ip nexthop show id 14 dev lo scope
host proto zebra id 15 dev ens33 scope host proto zebra id 16 dev ens36
scope host proto zebra id 17 dev ens37 scope host proto zebra id 18 dev
ens38 scope host proto zebra id 19 dev ens33 scope link proto zebra id
21 dev ens36 scope link proto zebra id 23 dev ens37 scope link proto
zebra id 25 dev ens38 scope link proto zebra id 26 dev lo3 scope link
proto zebra id 30 blackhole proto zebra id 31 blackhole proto zebra id
32 via 192.168.1.1 dev ens33 scope link proto zebra id 36 dev ens37
scope host proto zebra id 37 dev lo scope host proto zebra id 38 dev
ens38 scope host proto zebra root@FRR01:/opt/Kitts/frr/9.0.2# ip nexthop
show vrf red id 18 dev ens38 scope host proto zebra id 25 dev ens38
scope link proto zebra id 38 dev ens38 scope host proto zebra
root@FRR01:/opt/Kitts/frr/9.0.2# ip route list 10.0.0.0/30 dev ens37
proto kernel scope link src 10.0.0.1 10.0.1.0/30 nhid 38 dev ens38 proto
bgp metric 20 root@FRR01:/opt/Kitts/frr/9.0.2# ip route show table local
local 10.0.0.1 dev ens37 proto kernel scope host src 10.0.0.1 broadcast
10.0.0.3 dev ens37 proto kernel scope link src 10.0.0.1 local 10.100.0.1
dev lo proto kernel scope host src 10.100.0.1 broadcast 10.100.0.1 dev
lo proto kernel scope link src 10.100.0.1 local 127.0.0.0/8 dev lo proto
kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel
scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel
scope link src 127.0.0.1 root@FRR01:/opt/Kitts/frr/9.0.2# ip route show
vrf red blackhole default proto static metric 20 10.0.0.0/30 nhid 36 dev
ens37 proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link
src 10.0.1.1 10.100.0.1 nhid 37 dev lo proto bgp metric 20
root@FRR01:/opt/Kitts/frr/9.0.2# ip route show table red blackhole
default proto static metric 20 10.0.0.0/30 nhid 36 dev ens37 proto bgp
metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
local 10.0.1.1 dev ens38 proto kernel scope host src 10.0.1.1 broadcast
10.0.1.3 dev ens38 proto kernel scope link src 10.0.1.1 10.100.0.1 nhid
37 dev lo proto bgp metric 20 root@FRR01:/opt/Kitts/frr/9.0.2# ip route
show vrf red blackhole default proto static metric 20 10.0.0.0/30 nhid
36 dev ens37 proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel
scope link src 10.0.1.1 10.100.0.1 nhid 37 dev lo proto bgp metric 20
root@FRR01:/opt/Kitts/frr/9.0.2# ip rule list 0: from all lookup local
1000: from all lookup [l3mdev-table] 32766: from all lookup main 32767:
from all lookup default root@FRR01:/opt/Kitts/frr/9.0.2# Non-working
scenario with FRR 10.0: ||root@FRR01:/# ip nexthop show id 2 dev lo0 scope link proto zebra id 4
dev lo1 scope link proto zebra id 6 dev lo2 scope link proto zebra id 8
dev lo3 scope link proto zebra id 10 dev ens36 scope host proto zebra id
17 dev ens37 scope host proto zebra id 18 dev ens38 scope host proto
zebra id 19 dev lo scope host proto zebra id 20 dev ens33 scope host
proto zebra id 21 blackhole proto zebra id 22 blackhole proto zebra id
24 via 192.168.1.1 dev ens33 scope link proto zebra id 32 dev ens33
scope link proto zebra id 34 dev lo scope host proto zebra id 36 dev red
scope host proto zebra root@FRR01:/# ip nexthop show vrf red id 18 dev
ens38 scope host proto zebra id 25 dev ens38 scope link proto zebra|
|root@FRR01:/# ip route list|
|10.0.0.0/30 dev ens37 proto kernel scope link src 10.0.0.1 10.0.1.0/30
nhid 36 dev red proto bgp metric 20 root@FRR01:/# ip route show table
local local 10.0.0.1 dev ens37 proto kernel scope host src 10.0.0.1
broadcast 10.0.0.3 dev ens37 proto kernel scope link src 10.0.0.1 local
10.100.0.1 dev lo proto kernel scope host src 10.100.0.1 broadcast
10.100.0.1 dev lo proto kernel scope link src 10.100.0.1 local
127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1
dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255
dev lo proto kernel scope link src 127.0.0.1 root@FRR01:/# ip route show
vrf red blackhole default proto static metric 20 10.0.0.0/30 nhid 34 dev
lo proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src
10.0.1.1 10.100.0.1 nhid 34 dev lo proto bgp metric 20 root@FRR01:/# ip
route show table red blackhole default proto static metric 20
10.0.0.0/30 nhid 34 dev lo proto bgp metric 20 10.0.1.0/30 dev ens38
proto kernel scope link src 10.0.1.1 local 10.0.1.1 dev ens38 proto
kernel scope host src 10.0.1.1 broadcast 10.0.1.3 dev ens38 proto kernel
scope link src 10.0.1.1 10.100.0.1 nhid 34 dev lo proto bgp metric 20
root@FRR01:/# ip route show vrf red blackhole default proto static
metric 20 10.0.0.0/30 nhid 34 dev lo proto bgp metric 20 10.0.1.0/30 dev
ens38 proto kernel scope link src 10.0.1.1 10.100.0.1 nhid 34 dev lo
proto bgp metric 20 root@FRR01:/# ip ru l 0: from all lookup local 1000:
from all lookup [l3mdev-table] 32766: from all lookup main 32767: from
all lookup default|
As you can see the difference is how FRRouting is installing the routes:
9.0.2 and 9.1 is pointing towards ensXX and 10.x is pointing to lo or
red interfaces.
When is pointing to default VRF is pointing to interface lo. Then there
there is a L3 loop until the packet is reaching TTL 0.
I have no clue how it suppose to work correctly: pointing the routes to
VRF interface as 10.0 is doing or towards the route interface where is
installed,like ensXX, as 9.1 is doing.
It suppose that lo interface should reinject the traffic in the kernel
network stack or is this the normal behavior?
Kind regards,
Easynet
On 17.05.2024 14:52, Diederik de Haas wrote:
Control: tag -1 moreinfo
On 15 May 2024 16:08:27 +0200 Development EasyNet<[email protected]> wrote:
Package: linux-image
Version: 6.6.15-2 and 6.7.12-1
I'm facing for some time a strange behavior of the route-leak. It happen
on both IPv4 and IPv6.
Configuration used: Debian Trixie, Kernel 6.7.12 with FRRouting 10.1 - git
VRF: internet
Default: just local management
Sid recently got a 6.8.9 kernel, can you test whether that fixes the issue?
--
Development @EasyNet
Web: www.easynet.dev <https://www.easynet.dev>
