Re: Linux IPv6 routing strange behaviour
On Wed, 14 Aug 2013, Max Tulyev wrote: What is the soultion? There are *MILLIONS* of flows in the backbone... The solution is not to use a flow routing platform in the core. This lesson was learnt at the end of the 90ties. So until the linux ipv6 forwarding code is fixed to do stateless forwarding, it's just not suited for your application. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Linux IPv6 routing strange behaviour
On 8/14/13 14:30 , Max Tulyev wrote: OMFG 8-| What is the soultion? There are *MILLIONS* of flows in the backbone... I'd try increasing the routes max_size limit as a first step. If you're running heavy traffic and/or full routes through a linux box, you're going to need to do some tuning and trial-and-error testing - a lot of default parameters are not optimized for a router role. You may also want to decrease the gc_interval, and poke around at some of the other knobs there. -e On 14.08.13 21:10, Hannes Frederic Sowa wrote: On Wed, Aug 14, 2013 at 08:00:49PM +0300, Max Tulyev wrote: On 14.08.13 13:59, Hannes Frederic Sowa wrote: If a packet is delivered to a destination, we clone the routing entry and reinsert it back into the fib trie. Does it mean the original route is keept or deleted? Does it do for EVERY packet, i.e. EVERY packet generates a (temporary) FIB entry??? Every flow does generate a FIB entry, yes. As soon as the fib entry is installed it will get reused or discarded as soon as the GC decides so. Greetings, Hannes
Re: Linux IPv6 routing strange behaviour
On Wed, Aug 14, 2013 at 09:30:01PM +0300, Max Tulyev wrote: > OMFG 8-| > > What is the soultion? There are *MILLIONS* of flows in the backbone... Discard them fast enough. ;) The garbage collector is called if you reach the limit (in the code path). But sometimes it cannot free enough entries. In that moment you should see a warning message in dmesg which should point you to route/max_size. If that is not the case, I guess you could also have another problem. Also read up on the other knobs in /proc/sys/net/ipv6/route/*gc*. You should handle a fairly decent throughput but as soon as a lot of udp/dns packets traverse the router it might get a bit critical. But that is the case with every flow based routing architecture. Greetings, Hannes
Re: Linux IPv6 routing strange behaviour
OMFG 8-| What is the soultion? There are *MILLIONS* of flows in the backbone... On 14.08.13 21:10, Hannes Frederic Sowa wrote: > On Wed, Aug 14, 2013 at 08:00:49PM +0300, Max Tulyev wrote: >> On 14.08.13 13:59, Hannes Frederic Sowa wrote: >>> If a packet is delivered to a destination, we clone the routing entry and >>> reinsert it back into the fib trie. >> >> Does it mean the original route is keept or deleted? >> >> Does it do for EVERY packet, i.e. EVERY packet generates a (temporary) >> FIB entry??? > > Every flow does generate a FIB entry, yes. As soon as the fib entry is > installed it will get reused or discarded as soon as the GC decides so. > > Greetings, > > Hannes > >
Re: Linux IPv6 routing strange behaviour
On Wed, Aug 14, 2013 at 08:00:49PM +0300, Max Tulyev wrote: > On 14.08.13 13:59, Hannes Frederic Sowa wrote: > > If a packet is delivered to a destination, we clone the routing entry and > > reinsert it back into the fib trie. > > Does it mean the original route is keept or deleted? > > Does it do for EVERY packet, i.e. EVERY packet generates a (temporary) > FIB entry??? Every flow does generate a FIB entry, yes. As soon as the fib entry is installed it will get reused or discarded as soon as the GC decides so. Greetings, Hannes
Re: Linux IPv6 routing strange behaviour
On 14.08.13 13:59, Hannes Frederic Sowa wrote: > If a packet is delivered to a destination, we clone the routing entry and > reinsert it back into the fib trie. Does it mean the original route is keept or deleted? Does it do for EVERY packet, i.e. EVERY packet generates a (temporary) FIB entry???
Re: Linux IPv6 routing strange behaviour
On 14.08.13 17:09, Jeroen Massar wrote: >>> On the same box? Are they using the same routing table? I am fairly >>> confident that will end up in a fight. >> >> No! Some boxes have Quagga, some - Bird, not together of course. > > And are these boxes interconnected and are they thus possibly forwarding > the information to each other? Sure. It is interconnected, and routes are distributed from route reflectors. >> The prefixes from the global IPv6 routing table. It should be one >> routing entry per route, but have the lot of totally same route strings. > > Your view on 'global' will be different than that from any other person > depending on the peers you have, configuration settings and lots lots more. :) > People cause bugs? And if the path does not contain repeated ASNs it > will nicely loop till something stops it. But how it can affect to SAME routes duplication in the FIB? >> Which logs can explain something? > > The ones on your hosts? The ones generated by the tools that apparently > break. Nothing in /var/log/messages, Quagga/BIRD shows nothing interesting as well. No errors, no warnings.
Re: Linux IPv6 routing strange behaviour
On 2013-08-14 12:58 , Max Tulyev wrote: > On 14.08.13 13:39, Jeroen Massar wrote: >>> I see the strange behaviour of my Linux routers. There are quagga and >>> bird with IPv6 BGP full view. >> >> On the same box? Are they using the same routing table? I am fairly >> confident that will end up in a fight. > > No! Some boxes have Quagga, some - Bird, not together of course. And are these boxes interconnected and are they thus possibly forwarding the information to each other? >>> Quagga/bird reports about 13500 prefixes, >>> but route table constantly grow up to 10 routes and more. >> >> Which prefixes, and who/what originates them? > > The prefixes from the global IPv6 routing table. It should be one > routing entry per route, but have the lot of totally same route strings. Your view on 'global' will be different than that from any other person depending on the peers you have, configuration settings and lots lots more. Without details, little that can be stated though. Note that my teeny Quagga box receives multiple full feeds, though does not install them in a FIB, and has been running fine for over 10 years (except for some strange lockup bug once in a while, likely caused by multiple connections appearing/disappearing at the same time or some other odd and non-easily-reproduceable race condition) >> Are you maybe causing a routing loop towards yourself? > > I checked - no, but why a lot of routing entries, even if loop? People cause bugs? And if the path does not contain repeated ASNs it will nicely loop till something stops it. >>> Did anyone see something like that? How I can fix it? >> >> Providing outputs/log files would be a good start for people to look at it. > > Which logs can explain something? The ones on your hosts? The ones generated by the tools that apparently break. Of course it depends all on how you configure things, nobody can guess what your setup looks like or how it is acting. > P.S. It seems if there is more IPv6 traffic - there is faster growth of > routing table. One would hope so. Though at one point it will slow down, till companies/end-sites start getting PI prefixes... Greets, Jeroen
Re: Linux IPv6 routing strange behaviour
On Wed, Aug 14, 2013 at 01:49:23PM +0300, Max Tulyev wrote: > /proc/sys/net/ipv6/route/max_size - 10. Route table grows to 10 > entries, and then drops to almost zero. Then cycle again. > > But it should not be more than 14000, if mean both local and global > routes... Try increasing it more. Perhaps the errors there bring quagga into a state where it flushes the routes and tries to inject them again. If a packet is delivered to a destination, we clone the routing entry and reinsert it back into the fib trie. This is needed because we need to track pmtu and other stuff in this routing node. In contrast, IPv4 is using nexthop exceptions here which store this information in hash tables placed right to the statically configured fib_node. I would like to have this scheme one day for IPv6, too, because it enabled to do the linux kernel a lot more sharing between the nh-exceptions. But in contradiction to common believe, there is no routing cache as there was once for IPv4. ;) Greetings, Hannes
Re: Linux IPv6 routing strange behaviour
On 14.08.13 13:39, Jeroen Massar wrote: >> I see the strange behaviour of my Linux routers. There are quagga and >> bird with IPv6 BGP full view. > > On the same box? Are they using the same routing table? I am fairly > confident that will end up in a fight. No! Some boxes have Quagga, some - Bird, not together of course. >> Quagga/bird reports about 13500 prefixes, >> but route table constantly grow up to 10 routes and more. > > Which prefixes, and who/what originates them? The prefixes from the global IPv6 routing table. It should be one routing entry per route, but have the lot of totally same route strings. > Are you maybe causing a routing loop towards yourself? I checked - no, but why a lot of routing entries, even if loop? >> Did anyone see something like that? How I can fix it? > > Providing outputs/log files would be a good start for people to look at it. Which logs can explain something? P.S. It seems if there is more IPv6 traffic - there is faster growth of routing table.
Re: Linux IPv6 routing strange behaviour
/proc/sys/net/ipv6/route/max_size - 10. Route table grows to 10 entries, and then drops to almost zero. Then cycle again. But it should not be more than 14000, if mean both local and global routes... On 14.08.13 13:36, Hannes Frederic Sowa wrote: > On Wed, Aug 14, 2013 at 01:33:41PM +0300, Max Tulyev wrote: >> Hi All, >> >> I see the strange behaviour of my Linux routers. There are quagga and >> bird with IPv6 BGP full view. Quagga/bird reports about 13500 prefixes, >> but route table constantly grow up to 10 routes and more. Some >> routes are duplicated 2,3,5 and even up to 20 times :( And after some >> time routing stops working completely, number of routes drops to about >> zero, and start to grow again. >> >> Did anyone see something like that? How I can fix it? > > cat /proc/sys/net/ipv6/route/max_size? > > Maybe you have to increase this value. Didn't dmesg show something? > > Greetings, > > Hannes > >
Re: Linux IPv6 routing strange behaviour
On 2013-08-14 12:33, Max Tulyev wrote: > Hi All, > > I see the strange behaviour of my Linux routers. There are quagga and > bird with IPv6 BGP full view. On the same box? Are they using the same routing table? I am fairly confident that will end up in a fight. > Quagga/bird reports about 13500 prefixes, > but route table constantly grow up to 10 routes and more. Which prefixes, and who/what originates them? Are you maybe causing a routing loop towards yourself? > Some > routes are duplicated 2,3,5 and even up to 20 times :( And after some > time routing stops working completely, number of routes drops to about > zero, and start to grow again. You are aware that Linux is mostly meant as an end-user/server system, not as a routing platform? There are a LOT of scalability problems in the routing code as it was not designed for large scalability... > Did anyone see something like that? How I can fix it? Providing outputs/log files would be a good start for people to look at it. Greets, Jeroen
Re: Linux IPv6 routing strange behaviour
On Wed, Aug 14, 2013 at 01:33:41PM +0300, Max Tulyev wrote: > Hi All, > > I see the strange behaviour of my Linux routers. There are quagga and > bird with IPv6 BGP full view. Quagga/bird reports about 13500 prefixes, > but route table constantly grow up to 10 routes and more. Some > routes are duplicated 2,3,5 and even up to 20 times :( And after some > time routing stops working completely, number of routes drops to about > zero, and start to grow again. > > Did anyone see something like that? How I can fix it? cat /proc/sys/net/ipv6/route/max_size? Maybe you have to increase this value. Didn't dmesg show something? Greetings, Hannes
Linux IPv6 routing strange behaviour
Hi All, I see the strange behaviour of my Linux routers. There are quagga and bird with IPv6 BGP full view. Quagga/bird reports about 13500 prefixes, but route table constantly grow up to 10 routes and more. Some routes are duplicated 2,3,5 and even up to 20 times :( And after some time routing stops working completely, number of routes drops to about zero, and start to grow again. Did anyone see something like that? How I can fix it?
Re: Amount of announced IPv4-space by ASN not announcing IPv6?
Ignatios Souvatzis wrote: >On Tue, Aug 13, 2013 at 08:49:54PM +0200, Martin Millnert wrote: > >> We still have the last big problem with access enablement (how many >> NRENs have member universities with access-enabled IPv6?), and CPEs. > >In Germany, about 1.01 or 2.01 (the .01 being my part of my >department), >to my knowledge. > > -is There are more. Maybe not with full coverage of access ports but things like WiFi are v6-enabled for quite some more. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Amount of announced IPv4-space by ASN not announcing IPv6?
On Tue, Aug 13, 2013 at 08:49:54PM +0200, Martin Millnert wrote: > We still have the last big problem with access enablement (how many > NRENs have member universities with access-enabled IPv6?), and CPEs. In Germany, about 1.01 or 2.01 (the .01 being my part of my department), to my knowledge. -is
