Re: IPv6 BGP & kernel 4.19 (and upto 5.10.46)

2021-08-25 Thread Nico Schottelius
Hey Oliver, Oliver writes: > [...] > Why is still the default value of net.ipv6.route.max_size still 4096? > Compared to IPv4 value: > net.ipv4.route.max_size = 2147483647 > > Has someone done more research on this topic? I believe this is a question that should be asked on the LKML or linux-n

Re: IPv6 BGP & kernel 4.19 (and upto 5.10.46)

2021-08-25 Thread Oliver
Hello, back again on this topic. This problem is still not completely fixed with Debian Bullseye and kernel 5.10.46. The workaround is still: net.ipv6.route.max_size = 40 net.ipv6.route.gc_thresh = 102400 On https://bird.network.cz/pipermail/bird-users/2020-March/014406.html is mentioned th

Re: IPv6 BGP & kernel 4.19

2020-09-25 Thread Oliver
Hello, today I monitored the numbers and did more testing. The jitter starts, when $6 (total number of routes alloced) in /proc/net/rt6_stats is higher then the value of net.ipv6.route.gc_thresh. I proveked this by running: ip -6 route |egrep "^[0-9a-f]{1,4}:"|awk '{ print $1; }'|sed "s#/.*##"|x

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread Oliver
On Thu, 24 Sep 2020, Clément Guivy wrote: > On 24/09/2020 14:37, Oliver wrote: > > Hello, > > > > after upgrading to debian buster with kernel 4.19 we also had problems. > > How filled is your route cache compared to the sysctl treshold? See the > (hex) value with : > cut -d'' -f 6 /proc/net/rt6

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread Clément Guivy
On 24/09/2020 14:37, Oliver wrote: Hello, after upgrading to debian buster with kernel 4.19 we also had problems. How filled is your route cache compared to the sysctl treshold? See the (hex) value with : cut -d'' -f 6 /proc/net/rt6_stats Do you get a "Network is unreachable" error at some

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread Benedikt Neuffer
Hi all, On 24.09.20 17:03, micah anderson wrote: > Oliver writes: > >> Hello, >> >> after upgrading to debian buster with kernel 4.19 we also had problems. >> >> By adjusting net.ipv6.route.max_size we have fixed the following messages: >> watchdog: BUG: soft lockup - CPU#X stuck for 22s! >> an

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread micah anderson
Oliver writes: > Hello, > > after upgrading to debian buster with kernel 4.19 we also had problems. > > By adjusting net.ipv6.route.max_size we have fixed the following messages: > watchdog: BUG: soft lockup - CPU#X stuck for 22s! > and > ixgbe :02:00.0 ens2fX: initiating reset due to tx tim

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread Oliver
On Thu, 24 Sep 2020, Frederik Kriewitz wrote: > On Thu, Sep 24, 2020 at 2:47 PM Oliver wrote: > > @Frederik Kriewitz: What did you do fix that problem? > > We're not having jitter issues with debians 4.19 kernel: > > 100 packets transmitted, 100 received, 0% packet loss, time 99143ms > rtt min/

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread Frederik Kriewitz
On Thu, Sep 24, 2020 at 2:47 PM Oliver wrote: > @Frederik Kriewitz: What did you do fix that problem? We're not having jitter issues with debians 4.19 kernel: 100 packets transmitted, 100 received, 0% packet loss, time 99143ms rtt min/avg/max/mdev = 0.138/0.174/0.236/0.015 ms With these setting

Re: IPv6 BGP & kernel 4.19

2020-09-24 Thread Oliver
Hello, after upgrading to debian buster with kernel 4.19 we also had problems. By adjusting net.ipv6.route.max_size we have fixed the following messages: watchdog: BUG: soft lockup - CPU#X stuck for 22s! and ixgbe :02:00.0 ens2fX: initiating reset due to tx timeout But we still had a lot of

Re: IPv6 BGP & kernel 4.19

2020-03-29 Thread Alarig Le Lay
Hi, Here `net.ipv6.route.gc_thresh = -1` seems to be sufficent. Thanks for the idea! Alarig On lun. 16 mars 22:10:28 2020, Clément Guivy wrote: > Thanks. > > I found a solution which seems to be working so far, with regular Debian > 4.19 kernel, on my 2 edge routers. > > I set both net.ipv6.g

Re: IPv6 BGP & kernel 4.19

2020-03-16 Thread Clément Guivy
Thanks. I found a solution which seems to be working so far, with regular Debian 4.19 kernel, on my 2 edge routers. I set both net.ipv6.gc_thresh and max_size to 131072, the reasoning behind that is to have this limit above the number of routes in the full view, so that gc is not triggered t

Re: IPv6 BGP & kernel 4.19

2020-03-16 Thread Baptiste Jonglez
FYI, babeld seems to be affected by this same bug: https://github.com/jech/babeld/issues/50 The net.ipv6.route.max_size workaround is also mentioned there. Baptiste On 26-02-20, Basil Fillan wrote: > Hi, > > We've also experienced this after upgrading a few routers to Debian Buster. > With a k

Re: IPv6 BGP & kernel 4.19

2020-02-26 Thread Basil Fillan
Hi, We've also experienced this after upgrading a few routers to Debian Buster. With a kernel bisect we found that a bug was introduced in the following commit: 3b6761d18bc11f2af2a6fc494e9026d39593f22c This bug was still present in master as of a few weeks ago. It appears entries are added

Re: IPv6 BGP & kernel 4.19

2020-02-26 Thread Clément Guivy
Hi, did anyone find a solution or workaround regarding this issue? Considering a router use case. I have looked at rt6_stats, total route count is around 78k (full view), and around 4100 entries in the cache at the moment on my first router (forwarding a few Mb/s) and around 2500 entries on my s

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Alarig Le Lay
We agree then, and I act as a router on all those machines. Le 3 décembre 2019 19:27:11 GMT+01:00, Vincent Bernat a écrit : >This is the result of PMTUd. But when you are a router, you don't need >to do that, so it's mostly a problem for end hosts. > >On December 3, 2019 7:05:49 PM GMT+01:00, Al

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Alarig Le Lay
On 03/12/2019 14:16, Vincent Bernat wrote: > The information needs to be stored somewhere. Why has it to be stored? It’s not really my problem if someone else has a non-stantard MTU and can’t do TCP-MSS or PMTUd. -- Alarig

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Vincent Bernat
❦ 3 décembre 2019 12:48 +01, Alarig Le Lay : >> It's not unexpected. A cache entry is for a /128. > > When I’m routing 80k prefixes I don’t want to have n /128 routes because > someone doesn’t have 1500 of MTU. Is their a way to disable this > behaviour? I don't think there is. The information

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Alarig Le Lay
On 03/12/2019 11:58, Vincent Bernat wrote: > It's not unexpected. A cache entry is for a /128. When I’m routing 80k prefixes I don’t want to have n /128 routes because someone doesn’t have 1500 of MTU. Is their a way to disable this behaviour? -- Alarig

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Vincent Bernat
❦ 3 décembre 2019 11:46 +01, Alarig Le Lay : > So, I have more routes in cache than in FIB on my two core routers, I’m > pretty sure there is a bug there :p It's not unexpected. A cache entry is for a /128. > I have less routes in cache on 4.14 kernels but more traffic. > > I don’t know which

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Alarig Le Lay
On mar. 3 déc. 09:40:31 2019, Vincent Bernat wrote: > So, there is 0x56 entries in the cache. Isn't that clear? :) > > https://elixir.bootlin.com/linux/latest/source/net/ipv6/route.c#L6006 I did a quick test on some routers: core01-arendal, no fullview, on my own ASN, no so much traffic, using

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Vincent Bernat
❦ 3 décembre 2019 08:56 +01, Alarig Le Lay : >> Just to be clear: I did forget this fact and therefore my initial >> recommendation to increase max_size with more than 4096 active hosts >> does not apply anymore (as long as you have a 4.2+ kernel). Keep the >> default value and watch `/proc/net/

Re: IPv6 BGP & kernel 4.19

2019-12-03 Thread Alarig Le Lay
On 02/12/2019 23:04, Vincent Bernat wrote: > Just to be clear: I did forget this fact and therefore my initial > recommendation to increase max_size with more than 4096 active hosts > does not apply anymore (as long as you have a 4.2+ kernel). Keep the > default value and watch `/proc/net/rt6_stats

Re: IPv6 BGP & kernel 4.19

2019-12-02 Thread Vincent Bernat
❦ 2 décembre 2019 22:48 +01, Vincent Bernat : > Also, from 4.2, the cache entries are only created for exceptions (PMTU > notably). So, in fact, the initial value should be mostly safe. You can > monitor it with `/proc/net/rt6_stats`. This is the before last value. If > you can share what you ha

Re: IPv6 BGP & kernel 4.19

2019-12-02 Thread Vincent Bernat
❦ 2 décembre 2019 21:58 +01, Alarig Le Lay : >> For IPv6, this is the size of the routing cache. If you have more than >> 4096 active hosts, Linux will aggressively try to run garbage >> collection, eating CPU. In this case, increase both >> net.ipv6.route.max_size and net.ipv6.route.gc_thresh.

Re: IPv6 BGP & kernel 4.19

2019-12-02 Thread Alarig Le Lay
Hi Vincent, On lun. 2 déc. 21:38:21 2019, Vincent Bernat wrote: > For IPv6, this is the size of the routing cache. If you have more than > 4096 active hosts, Linux will aggressively try to run garbage > collection, eating CPU. In this case, increase both > net.ipv6.route.max_size and net.ipv6.rou

Re: IPv6 BGP & kernel 4.19

2019-12-02 Thread Vincent Bernat
❦ 1 décembre 2019 19:20 +01, Clément Guivy : > Hi, that's good news. One thing that still confuses me though is that > the default values for these settings are the same in Debian 9 (4.9 > kernel) and Debian 10 (4.19 kernel), so I would expect the behaviour > to be the same between both versions

Re: IPv6 BGP & kernel 4.19

2019-12-02 Thread Andrew Hearn
On 01/12/2019 18:20, Clément Guivy wrote: > On 01/12/2019 13:43, Frederik Kriewitz wrote: >> This is our current suspicion too. neighbours and routes are well >> below 4096 in our case. We also had to adjust >> net.ipv6.neigh.default.gc_thresh1/2/3. Since the adjustment it's been >> working fine. >

Re: IPv6 BGP & kernel 4.19

2019-12-01 Thread Clément Guivy
On 01/12/2019 13:43, Frederik Kriewitz wrote: This is our current suspicion too. neighbours and routes are well below 4096 in our case. We also had to adjust net.ipv6.neigh.default.gc_thresh1/2/3. Since the adjustment it's been working fine. Hi, that's good news. One thing that still confuses

Re: IPv6 BGP & kernel 4.19

2019-12-01 Thread Frederik Kriewitz
On Sun, Dec 1, 2019 at 12:57 PM Daniel Suchy wrote: > One idea that comes in my mind is default kernel limit for IPv6 routes > in memory (sysctl net.ipv6.route.max_size); and such default is quite > low for fullbgp/DFZ IPv6 deployments and it's still set to 4096 on > Debian/Buster with stock kerne

Re: IPv6 BGP & kernel 4.19

2019-12-01 Thread Daniel Suchy
Hello, I'm running bird 1.6.x branch (packages from Debian/Buster; currently 1.6.6) on recent 4.19 custom-build kernels without any issues (on armhf hardware). My BGP sessions are carrying only few routes (default + some more specifics). One idea that comes in my mind is default kernel limit for

Re: IPv6 BGP & kernel 4.19

2019-12-01 Thread Benedikt Neuffer
Hi Frederik, On 30.11.19 23:31, Frederik Kriewitz wrote: > On Sat, Nov 30, 2019 at 12:26 PM Benedikt Neuffer > wrote: > Which NICs are you using? We are using Intel X520. Regards, Benedikt -- Karlsruher Institut für Technologie (KIT) Steinbuch Centre for Computing (SCC) Benedikt Neuffer Net

Re: IPv6 BGP & kernel 4.19

2019-11-30 Thread Alarig Le Lay
On sam. 30 nov. 23:50:48 2019, Alarig Le Lay wrote: > We are using “Intel Corporation 82576 Gigabit Network Connection” NICs. And “Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet”, sorry I forgot this box. -- Alarig

Re: IPv6 BGP & kernel 4.19

2019-11-30 Thread Alarig Le Lay
On sam. 30 nov. 23:31:39 2019, Frederik Kriewitz wrote: > We don't know if this might be NIC related yet. We're seeing it happen > with Intel X710 NICs (With all offloading features disabled). Which > NICs are you using? We are using “Intel Corporation 82576 Gigabit Network Connection” NICs. --

Re: IPv6 BGP & kernel 4.19

2019-11-30 Thread Frederik Kriewitz
On Sat, Nov 30, 2019 at 12:26 PM Benedikt Neuffer wrote: > as far as I see one need some traffic to reproduce the issue. Without > traffic I haven't seen the issue. Yes, we saw this behaviour too using the buster kernel. It seems to be traffic and/or neighbours related. Forwarding itself seems to

Re: IPv6 BGP & kernel 4.19

2019-11-30 Thread Alarig Le Lay
I saw it in production with ~20 VMs, but I don’t know how much is needed to trigger it. On sam. 30 nov. 11:43:29 2019, Stefan Jakob wrote: > Can anyone provide test configs? > > Is it testable inside two or three VMs? > > Could offer 5.3.X tests here. > > On Sat, Nov 23, 2019 at 6:48 PM Alarig

Re: IPv6 BGP & kernel 4.19

2019-11-30 Thread Benedikt Neuffer
Hi all, On 30.11.19 11:43, Stefan Jakob wrote: > Can anyone provide test configs? > > Is it testable inside two or three VMs? > > Could offer 5.3.X tests here. > > On Sat, Nov 23, 2019 at 6:48 PM Alarig Le Lay wrote: >> >> On jeu. 21 nov. 18:12:17 2019, Ondrej Zajicek wrote: >>> Perhaps try ke

Re: IPv6 BGP & kernel 4.19

2019-11-30 Thread Stefan Jakob
Can anyone provide test configs? Is it testable inside two or three VMs? Could offer 5.3.X tests here. On Sat, Nov 23, 2019 at 6:48 PM Alarig Le Lay wrote: > > On jeu. 21 nov. 18:12:17 2019, Ondrej Zajicek wrote: > > Perhaps try kernel 5.2.x or 5.3.x from buster-backports? > > I’m very interest

Re: IPv6 BGP & kernel 4.19

2019-11-23 Thread Alarig Le Lay
On jeu. 21 nov. 18:12:17 2019, Ondrej Zajicek wrote: > Perhaps try kernel 5.2.x or 5.3.x from buster-backports? I’m very interested by test results from newer kernels than 5.0.x -- Alarig

Re: IPv6 BGP & kernel 4.19

2019-11-21 Thread Ondrej Zajicek
On Thu, Nov 21, 2019 at 04:09:24PM +, Andrew Hearn wrote: > > Without traffic through the box (all IPv6 prefixes filtered) the bgp > > sessions is stable. With traffic the bgp session dies after some time > > and ssh connections in the default table freezes. > > > > I did some packet captures

Re: IPv6 BGP & kernel 4.19

2019-11-21 Thread Alarig Le Lay
Hi, On 21/11/2019 17:46, Benedikt Neuffer wrote: > Hi Andrew, > > On 21.11.19 17:09, Andrew Hearn wrote: >> Sorry to bring up a fairly old thread... >> >> We believe we are seeing this problem too, since a Stretch->Buster >> upgrade - was there a solution to this? >> >> Thanks > > The problem st

Re: IPv6 BGP & kernel 4.19

2019-11-21 Thread Benedikt Neuffer
Hi Andrew, On 21.11.19 17:09, Andrew Hearn wrote: > Sorry to bring up a fairly old thread... > > We believe we are seeing this problem too, since a Stretch->Buster > upgrade - was there a solution to this? > > Thanks The problem still exists. We are still running on kernel 4.14.x. I had no time

Re: IPv6 BGP & kernel 4.19

2019-11-21 Thread Andrew Hearn
On 20/06/2019 17:13, Benedikt Neuffer wrote: > Hi, > > On 19.06.19 20:09, Alarig Le Lay wrote: >> Hi, >> >> On mer. 19 juin 09:10:53 2019, Robert Sander wrote: >>> Hi, >>> >>> our routers run on Debian stretch with bird 1.6.4 from >>> bird.network.cz/debian. >>> >>> Yesterday I tried kernel 4.19 f

Re: IPv6 BGP & kernel 4.19

2019-06-20 Thread Benedikt Neuffer
Hi, On 19.06.19 20:09, Alarig Le Lay wrote: > Hi, > > On mer. 19 juin 09:10:53 2019, Robert Sander wrote: >> Hi, >> >> our routers run on Debian stretch with bird 1.6.4 from >> bird.network.cz/debian. >> >> Yesterday I tried kernel 4.19 from backports.debian.org and ran into a >> weird issue with

Re: IPv6 BGP & kernel 4.19

2019-06-19 Thread Alarig Le Lay
Hi, On mer. 19 juin 09:10:53 2019, Robert Sander wrote: > Hi, > > our routers run on Debian stretch with bird 1.6.4 from > bird.network.cz/debian. > > Yesterday I tried kernel 4.19 from backports.debian.org and ran into a > weird issue with IPv6 BGP sessions: > > All Peerings reported "Error: H