Re: VRF Issue Since kernel 5
[ FYI: you should not 'top post' in responses to netdev; rather comment inline with the previous message ] On 9/12/19 7:50 AM, Gowen wrote: > > Hi David -thanks for getting back to me > > > > The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go > > out mgmt-vrf. correct? No - 10.24.65.203 10.25.65.203, so should hit the > route leak rule as below (if I've put the 10.24.64.0/24 subnet anywhere it is > a typo) > > vmAdmin@NETM06:~$ ip ro get 10.24.65.203 fibmatch > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > > I've added the 127/8 route - no difference. you mean address on mgmt-vrf right? > > The reason for what you might think is an odd design is that I wanted any > non-VRF aware users to be able to come in and run all commands in default > context without issue, while production and mgmt traffic was separated still > > DNS is now working as long as /etc/resolv.conf is populated with my DNS > servers - a lot of people would be using this on Azure which uses netplan, so > they'll have the same issue, is there documentation I could update or raise a > bug to check the systemd-resolve servers as well? That is going to be the fundamental system problem: handing DNS queries off to systemd is losing the VRF context of the process doing the DNS query.
Re: VRF Issue Since kernel 5
currently: vmAdmin@NETM06:~$ uname -r 5.0.0-1018-azure vmAdmin@NETM06:~$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" I don't keep a history of kernel versions on test but I noticed it had gone to kernel 5 and stopped working - I'm about 80% sure that happened at the same time - I'll try and dig out some logs today to see what I can find for you as Linux is fairly new to me Gareth From: David Ahern Sent: 11 September 2019 17:09 To: Gowen ; Alexis Bauvin ; mmann...@vyatta.att-mail.com Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/11/19 3:01 PM, Gowen wrote: > Hi all, > > It looks like ip vrf exec checks /etc/resolv.conf (found with strace -e > trace=file sudo ip vrf exec mgmt-vrf host www.google.co.uk &> > ~/straceFileOfVrfHost.txt) , but as I'm on an Azure machine using > netplan, this file isn't updated with DNS servers. I have added my DNS > server to resolv.conf and now can update the cache with "sudo ip vrf > exec sudo apt update", if I am correct (which I'm not sure about as not > really my area) then this might be affecting more than just me. > > Also still not able to fix the updating cache from global VRF - which > would cause bother in prod environment to others as well so think it > would be good to get an RCA for it? > > thanks for your help so far, has been really interesting. > > Gareth > > > > *From:* Gowen > *Sent:* 11 September 2019 13:48 > *To:* David Ahern ; Alexis Bauvin > ; mmann...@vyatta.att-mail.com > > *Cc:* netdev@vger.kernel.org > *Subject:* Re: VRF Issue Since kernel 5 > > yep no problem: > > Admin@NETM06:~$ sudo sysctl -a | grep l3mdev > net.ipv4.raw_l3mdev_accept = 1 > net.ipv4.tcp_l3mdev_accept = 1 > net.ipv4.udp_l3mdev_accept = 1 > > > The source of the DNS issue in the vrf exec command is something to do > with networkd managing the DNS servers, I can fix it by explicitly > mentioning the DNS server: > > systemd-resolve --status --no-page > > > > Link 4 (mgmt-vrf) > Current Scopes: none > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > > Link 3 (eth1) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > Link 2 (eth0) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > there is no DNS server when I use ip vrf exec command (tcpdump shows > only loopback traffic when invoked without my DNS sever explicitly > entered) - odd as mgmt-vrf isnt L3 device so thought it would pick up > eth0 DNS servers? > > I dont think this helps with my update cache traffic from global vrf > though on port 80 > Let's back up a bit: your subject line says vrf issue since kernel 5. Did you update / change the OS as well? ie., the previous version that worked what is the OS and kernel version? What is the OS and kernel version with the problem?
Re: VRF Issue Since kernel 5
Hi David -thanks for getting back to me The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go out mgmt-vrf. correct? No - 10.24.65.203 10.25.65.203, so should hit the route leak rule as below (if I've put the 10.24.64.0/24 subnet anywhere it is a typo) vmAdmin@NETM06:~$ ip ro get 10.24.65.203 fibmatch 10.24.65.0/24 via 10.24.12.1 dev eth0 I've added the 127/8 route - no difference. The reason for what you might think is an odd design is that I wanted any non-VRF aware users to be able to come in and run all commands in default context without issue, while production and mgmt traffic was separated still DNS is now working as long as /etc/resolv.conf is populated with my DNS servers - a lot of people would be using this on Azure which uses netplan, so they'll have the same issue, is there documentation I could update or raise a bug to check the systemd-resolve servers as well? Gareth From: David Ahern Sent: 11 September 2019 18:02 To: Gowen ; netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 At LPC this week and just now getting a chance to process the data you sent. On 9/9/19 8:46 AM, Gowen wrote: > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) > except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > Admin@NETM06:~$ ip route show > default via 10.24.12.1 dev eth0 > 10.0.0.0/8 via 10.24.12.1 dev eth1 > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 interesting route table. This is default VRF but you have route leaking through eth0 which is in mgmt-vrf. > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > default via 10.24.12.1 dev eth0 > unreachable default metric 4278198272 > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go out mgmt-vrf. correct? 10.24.65.203 should hit the route "10.24.65.0/24 via 10.24.12.1 dev eth0" for both default VRF and mgmt-vrf. 10.24.64.203 will NOT hit a route leak entry so traverse the VRF associated with the context of the command (mgmt-vrf or default). Is that intentional? (verify with: `ip ro get 10.24.64.203 fibmatch` and `ip ro get 10.24.64.203 vrf mgmt-vrf fibmatch`) > > > > The strange activity occurs when I enter the command “sudo apt update” as I > can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with > tcpdump) out eth0 but for the actual update traffic there is no activity: > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? > security.ubuntu.com. (48) > > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A > 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256) > > > > You can see that the update traffic is returned but is not accepted by the > stack and a RST is sent > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq > 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr > 0,nop,wscale 7], length 0 > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq > 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr > 0,nop,wscale 7], length 0 > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq > 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val > 81957103 ecr 2029365856,nop,wscale 7], length 0 > > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq > 2279624827, win 0, length 0 > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq > 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val > 2257624414 ecr 3833463674,nop,wscale 8], length 0
Re: VRF Issue Since kernel 5
At LPC this week and just now getting a chance to process the data you sent. On 9/9/19 8:46 AM, Gowen wrote: > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) > except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > Admin@NETM06:~$ ip route show > default via 10.24.12.1 dev eth0 > 10.0.0.0/8 via 10.24.12.1 dev eth1 > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 interesting route table. This is default VRF but you have route leaking through eth0 which is in mgmt-vrf. > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > default via 10.24.12.1 dev eth0 > unreachable default metric 4278198272 > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go out mgmt-vrf. correct? 10.24.65.203 should hit the route "10.24.65.0/24 via 10.24.12.1 dev eth0" for both default VRF and mgmt-vrf. 10.24.64.203 will NOT hit a route leak entry so traverse the VRF associated with the context of the command (mgmt-vrf or default). Is that intentional? (verify with: `ip ro get 10.24.64.203 fibmatch` and `ip ro get 10.24.64.203 vrf mgmt-vrf fibmatch`) > > > > The strange activity occurs when I enter the command “sudo apt update” as I > can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with > tcpdump) out eth0 but for the actual update traffic there is no activity: > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? > security.ubuntu.com. (48) > > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A > 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A > 91.189.88.174 (256) > > > > You can see that the update traffic is returned but is not accepted by the > stack and a RST is sent > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq > 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr > 0,nop,wscale 7], length 0 > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq > 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr > 0,nop,wscale 7], length 0 > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq > 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val > 81957103 ecr 2029365856,nop,wscale 7], length 0 > > > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq > 2279624827, win 0, length 0 > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq > 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val > 2257624414 ecr 3833463674,nop,wscale 8], length 0 > > > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq > 1465797257, win 0, length 0 > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > `ip vrf exec mgmt-vrf ` means that every IPv4 and IPv6 socket opened by is automatically bound to mgmt-vrf which causes route lookups to hit the mgmt-vrf table. Just running (without binding to any vrf) means no socket is bound to anything unless the command does a bind. In that case the routing lookups determine which egress device is used. Now the response comes back, if the ingress interface is a VRF then the socket lookup wants to match on a device. Now, a later response shows this for DNS lookups: isc-worker 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev
Re: VRF Issue Since kernel 5
On 9/9/19 10:28 AM, Alexis Bauvin wrote: > Also, your `unreachable default metric 4278198272` route looks odd to me. > New recommendation from FRR group. See https://www.kernel.org/doc/Documentation/networking/vrf.txt and search for 4278198272
Re: VRF Issue Since kernel 5
On 9/11/19 3:01 PM, Gowen wrote: > Hi all, > > It looks like ip vrf exec checks /etc/resolv.conf (found with strace -e > trace=file sudo ip vrf exec mgmt-vrf host www.google.co.uk &> > ~/straceFileOfVrfHost.txt) , but as I'm on an Azure machine using > netplan, this file isn't updated with DNS servers. I have added my DNS > server to resolv.conf and now can update the cache with "sudo ip vrf > exec sudo apt update", if I am correct (which I'm not sure about as not > really my area) then this might be affecting more than just me. > > Also still not able to fix the updating cache from global VRF - which > would cause bother in prod environment to others as well so think it > would be good to get an RCA for it? > > thanks for your help so far, has been really interesting. > > Gareth > > > > *From:* Gowen > *Sent:* 11 September 2019 13:48 > *To:* David Ahern ; Alexis Bauvin > ; mmann...@vyatta.att-mail.com > > *Cc:* netdev@vger.kernel.org > *Subject:* Re: VRF Issue Since kernel 5 > > yep no problem: > > Admin@NETM06:~$ sudo sysctl -a | grep l3mdev > net.ipv4.raw_l3mdev_accept = 1 > net.ipv4.tcp_l3mdev_accept = 1 > net.ipv4.udp_l3mdev_accept = 1 > > > The source of the DNS issue in the vrf exec command is something to do > with networkd managing the DNS servers, I can fix it by explicitly > mentioning the DNS server: > > systemd-resolve --status --no-page > > > > Link 4 (mgmt-vrf) > Current Scopes: none > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > > Link 3 (eth1) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > Link 2 (eth0) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > there is no DNS server when I use ip vrf exec command (tcpdump shows > only loopback traffic when invoked without my DNS sever explicitly > entered) - odd as mgmt-vrf isnt L3 device so thought it would pick up > eth0 DNS servers? > > I dont think this helps with my update cache traffic from global vrf > though on port 80 > Let's back up a bit: your subject line says vrf issue since kernel 5. Did you update / change the OS as well? ie., the previous version that worked what is the OS and kernel version? What is the OS and kernel version with the problem?
Re: VRF Issue Since kernel 5
Hi Gareth, Could you please also check that all the following are set to 1, I appreciate you've confirmed that the one for tcp is set to 1, and by default the one for raw is also set to 1: sudo sysctl -a | grep l3mdev If not, sudo sysctl net.ipv4.raw_l3mdev_accept=1 sudo sysctl net.ipv4.udp_l3mdev_accept=1 sudo sysctl net.ipv4.tcp_l3mdev_accept=1 Thanks Mike
Re: VRF Issue Since kernel 5
previously mentioned attchements From: Gowen Sent: 11 September 2019 12:19 To: David Ahern ; Alexis Bauvin Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 Hi there, Your perf command: isc-worker 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 Also I set all iptables to policy ACCEPT and flushed the rules, enabled forwarding, checked the sysctl settings are all '1'. I've looked at tracing DNS through the iptables and I see that DNS uses a loopback interface as source and destination - this would be odd on a Cisco box but having looked around this appears to be normal? I also gathered an strace of updating the package cache as well as a perf of the same command - will send if interested (is more verbose and not sure if the spam filter will block it) Gareth From: Gowen Sent: 11 September 2019 06:09 To: David Ahern ; Alexis Bauvin Cc: netdev@vger.kernel.org Subject: RE: VRF Issue Since kernel 5 Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change -Original Message- From: David Ahern Sent: 10 September 2019 17:36 To: Alexis Bauvin ; Gowen Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf sudo 25916 [000] 4124.775681: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/47817 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.775699: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/47817 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.776123: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/6 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.776125: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/6 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.776530: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 sudo 25916 [000] 4124.776532: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 sudo 25916 [000] 4124.776533: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.9/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 sudo 25916 [000] 4124.776533: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.9/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 sudo 25916 [000] 4124.776538: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/51581 -> 10.24.12.9/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 sudo 25916 [000] 4124.776539: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.9/51581 -> 10.24.12.9/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 http 25921 [001] 4124.841213: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/37649 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25921 [001] 4124.841216: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/37649 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 1
Re: VRF Issue Since kernel 5
Hi there, Your perf command: isc-worker 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 Also I set all iptables to policy ACCEPT and flushed the rules, enabled forwarding, checked the sysctl settings are all '1'. I've looked at tracing DNS through the iptables and I see that DNS uses a loopback interface as source and destination - this would be odd on a Cisco box but having looked around this appears to be normal? I also gathered an strace of updating the package cache as well as a perf of the same command - will send if interested (is more verbose and not sure if the spam filter will block it) Gareth From: Gowen Sent: 11 September 2019 06:09 To: David Ahern ; Alexis Bauvin Cc: netdev@vger.kernel.org Subject: RE: VRF Issue Since kernel 5 Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change -Original Message- From: David Ahern Sent: 10 September 2019 17:36 To: Alexis Bauvin ; Gowen Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf
RE: VRF Issue Since kernel 5
Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change -Original Message- From: David Ahern Sent: 10 September 2019 17:36 To: Alexis Bauvin ; Gowen Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf
Re: VRF Issue Since kernel 5
On 9/9/19 8:46 AM, Gowen wrote: > > I can run: > > > Admin@NETM06:~$ host www.google.co.uk > www.google.co.uk has address 172.217.169.3 > www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003 > > > but I get a timeout for: > > > sudo ip vrf exec mgmt-vrf host www.google.co.uk sudo perf record -e fib:* ip vrf exec mgmt-vrf host www.google.co.uk sudo perf script I am guessing the table is wrong for your setup, but maybe the output (or ordering of events) sheds some light on the problem.
Re: VRF Issue Since kernel 5
On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf
Re: VRF Issue Since kernel 5
Hi Alexis, I enabled the target TRACE and found that the packet is passing through the security table - which I thought was for SELinux only. As far as I can tell the config is working, is being seen by iptables nut for some reason is not getting accepted by the local process - which isn't right surely. Debugs below from TRACE for the 91.0.0.0/8 subnet for the updates Sep 10 13:50:37 NETM06 kernel: [442740.425992] TRACE: raw:PREROUTING:policy:2 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Sep 10 13:50:37 NETM06 kernel: [442740.426045] TRACE: filter:INPUT:rule:1 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Sep 10 13:50:37 NETM06 kernel: [442740.426060] TRACE: security:INPUT:rule:1 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Sep 10 13:50:37 NETM06 kernel: [442740.426108] TRACE: security:INPUT:policy:2 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Admin@NETM06:~$ sudo iptables -L PREROUTING -t raw -n -v Chain PREROUTING (policy ACCEPT 56061 packets, 5260K bytes) pkts bytes target prot opt in out source destination 296 16480 TRACE tcp -- mgmt-vrf * 91.0.0.0/8 0.0.0.0/0 ctstate RELATED,ESTABLISHED tcp spt:80 Chain INPUT (policy DROP 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 330 18260 ACCEPT tcp -- mgmt-vrf * 91.0.0.0/8 0.0.0.0/0ctstate RELATED,ESTABLISHED tcp spt:80 Admin@NETM06:~$ sudo iptables -L -t security -n -v --line-numbers Chain INPUT (policy ACCEPT 4190 packets, 371K bytes) num pkts bytes target prot opt in out source destination 1 248 13980 LOGall -- * * 91.0.0.0/8 0.0.0.0/0LOG flags 0 level 4 prefix "LOG-SECURITY" From: Gowen Sent: 09 September 2019 20:43 To: Alexis Bauvin Cc: netdev@vger.kernel.org Subject: RE: VRF Issue Since kernel 5 Hi alexis, I did this earlier today and no change. I’ll look at trying to see if the return traffic is hitting the INPUT table tomorrow with some conntrack rules and see if it hits any of those rules. If not then do you have any hints/techniques I can use to find the source of the issue? Gareth -Original Message- From: Alexis Bauvin Sent: 09 September 2019 13:02 To: Gowen Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 Hi, I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault. Could you try swapping the local and l3mdev rules? `ip rule del pref 0; ip rule add from all lookup local pref 1001` I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf. Alexis > Le 9 sept. 2019 à 12:53, Gowen a écrit : > > Hi Alexis, > > Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept > net.ipv4.tcp_l3mdev_accept = 1 > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org > curl: (6) Could not resolve host: kernel.org > > the failure to resolve is the same with all DNS lookups from any > process I've run > > The route is there from the guide I originally used, I can't remember > the purpose but I know I don't need it - I've removed it now and no > change > > Admin@NETM06:~$ ip rule show > 0: from all lookup local > 1000: from all lookup [l3mdev-table] > 32766: from all lookup main > 32767: from all lookup default > > I could switch the VRFs over, but this is a test-box and i have prod boxes on > this as well so not so keen on that if I can avoid it. > > From what I can speculate, because the TCP return traffic is met with an RST, > it looks like it may be something to do with iptables - but even if I set the > policy to ACCEPT and flush all the rules, the beh
RE: VRF Issue Since kernel 5
Hi alexis, I did this earlier today and no change. I’ll look at trying to see if the return traffic is hitting the INPUT table tomorrow with some conntrack rules and see if it hits any of those rules. If not then do you have any hints/techniques I can use to find the source of the issue? Gareth -Original Message- From: Alexis Bauvin Sent: 09 September 2019 13:02 To: Gowen Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 Hi, I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault. Could you try swapping the local and l3mdev rules? `ip rule del pref 0; ip rule add from all lookup local pref 1001` I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf. Alexis > Le 9 sept. 2019 à 12:53, Gowen a écrit : > > Hi Alexis, > > Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept > net.ipv4.tcp_l3mdev_accept = 1 > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org > curl: (6) Could not resolve host: kernel.org > > the failure to resolve is the same with all DNS lookups from any > process I've run > > The route is there from the guide I originally used, I can't remember > the purpose but I know I don't need it - I've removed it now and no > change > > Admin@NETM06:~$ ip rule show > 0: from all lookup local > 1000: from all lookup [l3mdev-table] > 32766: from all lookup main > 32767: from all lookup default > > I could switch the VRFs over, but this is a test-box and i have prod boxes on > this as well so not so keen on that if I can avoid it. > > From what I can speculate, because the TCP return traffic is met with an RST, > it looks like it may be something to do with iptables - but even if I set the > policy to ACCEPT and flush all the rules, the behaviour remains the same. > > Is it possible that the TCP stack isn't aware of the session (as is mapped to > wrong VRF internally or something to that effect) and is therefore sending > the RST? > > Gareth > From: Alexis Bauvin > Sent: 09 September 2019 10:28 > To: Gowen > Cc: netdev@vger.kernel.org > Subject: Re: VRF Issue Since kernel 5 > > Hi, > > There has been some changes regarding VRF isolation in Linux 5 IIRC, > namely proper isolation of the default VRF. > > Some things you may try: > > - looking at the l3mdev_accept sysctls (e.g. > `net.ipv4.tcp_l3mdev_accept`) > - querying stuff from the management vrf through `ip vrf exec vrf-mgmt > ` > e.g. `ip vrf exec vrf-mgmt curl kernel.org` >`ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` > - reversing your logic: default VRF is your management one, the other one is > for your > other boxes > > Also, your `unreachable default metric 4278198272` route looks odd to me. > > What are your routing rules? (`ip rule`) > > Alexis > > > Le 9 sept. 2019 à 09:46, Gowen a écrit : > > > > Hi there, > > > > Dave A said this was the mailer to send this to: > > > > > > I’ve been using my management interface in a VRF for several months now and > > it’s worked perfectly – I’ve been able to update/upgrade the packages just > > fine and iptables works excellently with it – exactly as I needed. > > > > > > Since Kernel 5 though I am no longer able to update – but the issue > > is quite a curious one as some traffic appears to be fine (DNS > > lookups use VRF correctly) but others don’t (updating/upgrading the > > packages) > > > > > > I have on this device 2 interfaces: > > Eth0 for management – inbound SSH, DNS, updates/upgrades > > Eth1 for managing other boxes (ansible using SSH) > > > > > > Link and addr info shown below: > > > > > > Admin@NETM06:~$ ip link show > > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode > > DEFAULT group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > 2: eth0: mtu 1500 qdisc mq master > > mgmt-vrf state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > 3: eth1: mtu 1500 qdisc mq state UP mode > > DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > 4: mgmt-vrf: mtu 65536 qdisc noqueue state UP > > mode DEFAULT group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > Admin@NETM06:~$ ip addr > > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group > > default q
Re: VRF Issue Since kernel 5
Hi, I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault. Could you try swapping the local and l3mdev rules? `ip rule del pref 0; ip rule add from all lookup local pref 1001` I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf. Alexis > Le 9 sept. 2019 à 12:53, Gowen a écrit : > > Hi Alexis, > > Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept > net.ipv4.tcp_l3mdev_accept = 1 > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org > curl: (6) Could not resolve host: kernel.org > > the failure to resolve is the same with all DNS lookups from any process I've > run > > The route is there from the guide I originally used, I can't remember the > purpose but I know I don't need it - I've removed it now and no change > > Admin@NETM06:~$ ip rule show > 0: from all lookup local > 1000: from all lookup [l3mdev-table] > 32766: from all lookup main > 32767: from all lookup default > > I could switch the VRFs over, but this is a test-box and i have prod boxes on > this as well so not so keen on that if I can avoid it. > > From what I can speculate, because the TCP return traffic is met with an RST, > it looks like it may be something to do with iptables - but even if I set the > policy to ACCEPT and flush all the rules, the behaviour remains the same. > > Is it possible that the TCP stack isn't aware of the session (as is mapped to > wrong VRF internally or something to that effect) and is therefore sending > the RST? > > Gareth > From: Alexis Bauvin > Sent: 09 September 2019 10:28 > To: Gowen > Cc: netdev@vger.kernel.org > Subject: Re: VRF Issue Since kernel 5 > > Hi, > > There has been some changes regarding VRF isolation in Linux 5 IIRC, namely > proper > isolation of the default VRF. > > Some things you may try: > > - looking at the l3mdev_accept sysctls (e.g. `net.ipv4.tcp_l3mdev_accept`) > - querying stuff from the management vrf through `ip vrf exec vrf-mgmt > ` > e.g. `ip vrf exec vrf-mgmt curl kernel.org` >`ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` > - reversing your logic: default VRF is your management one, the other one is > for your > other boxes > > Also, your `unreachable default metric 4278198272` route looks odd to me. > > What are your routing rules? (`ip rule`) > > Alexis > > > Le 9 sept. 2019 à 09:46, Gowen a écrit : > > > > Hi there, > > > > Dave A said this was the mailer to send this to: > > > > > > I’ve been using my management interface in a VRF for several months now and > > it’s worked perfectly – I’ve been able to update/upgrade the packages just > > fine and iptables works excellently with it – exactly as I needed. > > > > > > Since Kernel 5 though I am no longer able to update – but the issue is > > quite a curious one as some traffic appears to be fine (DNS lookups use VRF > > correctly) but others don’t (updating/upgrading the packages) > > > > > > I have on this device 2 interfaces: > > Eth0 for management – inbound SSH, DNS, updates/upgrades > > Eth1 for managing other boxes (ansible using SSH) > > > > > > Link and addr info shown below: > > > > > > Admin@NETM06:~$ ip link show > > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode > > DEFAULT group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > 2: eth0: mtu 1500 qdisc mq master > > mgmt-vrf state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > 3: eth1: mtu 1500 qdisc mq state UP mode > > DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > 4: mgmt-vrf: mtu 65536 qdisc noqueue state UP > > mode DEFAULT group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > Admin@NETM06:~$ ip addr > > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group > > default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > >valid_lft forever preferred_lft forever > > inet6 ::1/128 scope host > >valid_lft forever preferred_lft forever > > 2: eth0: mtu 1500 qdisc mq master > > mgmt-vrf state UP group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.10/24 brd 10.24
Re: VRF Issue Since kernel 5
Hi, There has been some changes regarding VRF isolation in Linux 5 IIRC, namely proper isolation of the default VRF. Some things you may try: - looking at the l3mdev_accept sysctls (e.g. `net.ipv4.tcp_l3mdev_accept`) - querying stuff from the management vrf through `ip vrf exec vrf-mgmt ` e.g. `ip vrf exec vrf-mgmt curl kernel.org` `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` - reversing your logic: default VRF is your management one, the other one is for your other boxes Also, your `unreachable default metric 4278198272` route looks odd to me. What are your routing rules? (`ip rule`) Alexis > Le 9 sept. 2019 à 09:46, Gowen a écrit : > > Hi there, > > Dave A said this was the mailer to send this to: > > > I’ve been using my management interface in a VRF for several months now and > it’s worked perfectly – I’ve been able to update/upgrade the packages just > fine and iptables works excellently with it – exactly as I needed. > > > Since Kernel 5 though I am no longer able to update – but the issue is quite > a curious one as some traffic appears to be fine (DNS lookups use VRF > correctly) but others don’t (updating/upgrading the packages) > > > I have on this device 2 interfaces: > Eth0 for management – inbound SSH, DNS, updates/upgrades > Eth1 for managing other boxes (ansible using SSH) > > > Link and addr info shown below: > > > Admin@NETM06:~$ ip link show > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode > DEFAULT group default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > 2: eth0: mtu 1500 qdisc mq master mgmt-vrf > state UP mode DEFAULT group default qlen 1000 > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > 3: eth1: mtu 1500 qdisc mq state UP mode > DEFAULT group default qlen 1000 > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > 4: mgmt-vrf: mtu 65536 qdisc noqueue state UP mode > DEFAULT group default qlen 1000 > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > Admin@NETM06:~$ ip addr > 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group > default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet 127.0.0.1/8 scope host lo >valid_lft forever preferred_lft forever > inet6 ::1/128 scope host >valid_lft forever preferred_lft forever > 2: eth0: mtu 1500 qdisc mq master mgmt-vrf > state UP group default qlen 1000 > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0 >valid_lft forever preferred_lft forever > inet6 fe80::222:48ff:fe07:ccad/64 scope link >valid_lft forever preferred_lft forever > 3: eth1: mtu 1500 qdisc mq state UP group > default qlen 1000 > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1 >valid_lft forever preferred_lft forever > inet6 fe80::222:48ff:fe07:c96c/64 scope link >valid_lft forever preferred_lft forever > 4: mgmt-vrf: mtu 65536 qdisc noqueue state UP > group default qlen 1000 > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) > except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > Admin@NETM06:~$ ip route show > default via 10.24.12.1 dev eth0 > 10.0.0.0/8 via 10.24.12.1 dev eth1 > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > default via 10.24.12.1 dev eth0 > unreachable default metric 4278198272 > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > The strange activity occurs when I enter the command “sudo apt update” as I > can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with > tcpdump) out eth0 but for the actual update traffic there is no activity: > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? > security.ubuntu.com. (48) > > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A > 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A > 91.189.88.174 (256) > > > > You can see that the update traffic is returned but is not accepted by the > stack and a RST is sent > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > tcpdump: verbose output suppressed, use -v or -vv for full protocol dec
