Re: Bird 2.0.0 memory leak

[Miłosz Oller]

Daemon.log:

Dec 16 08:29:42 r01 bird: Kernel dropped some netlink messages, will 
resync on next scan.



inż. Miłosz Oller

 W dniu 16.12.2017 o 00:44, Ondrej Zajicek pisze:


On Fri, Dec 15, 2017 at 03:42:03PM +, Rhys Williams wrote:

Hey,

I am also seeing this memory leak on Ubuntu 16.04, kernel 4.10.0-36.

The memory leak seems to occur when the scan timer lapses on the kernel 
protocol leaking about the full tables worth of memory each time until 
eventually bird is oom killed.

This patch should fix it.







smime.p7s
Description: Kryptograficzna sygnatura S/MIME

Re: Bird 2.0.0 memory leak

[Miłosz Oller]

Hi

After bird starts:

bird> show memory
BIRD memory usage
Routing tables:    598 MB
Route attributes:  161 MB
Protocols: 306 kB
Total: 759 MB


After next ~5mins:

bird> show memory
BIRD memory usage
Routing tables:   1534 MB
Route attributes:  161 MB
Protocols: 306 kB
Total:    1696 MB


Config with 17 sessions and one iBGP:

one of sessions:

protocol bgp bgp_atm_pl_backup {
    description "ATM Peering.PL Backup";
#    import all;
ipv4 { import all; };
ipv6 { import all; };
    ipv4 { export filter ipv4_export; };
    local as 198611;
    neighbor 212.91.2.1 as 24748;
    source address 212.91.1.42;
#    default bgp_local_pref 110;
}


Miłosz

W dniu 16.12.2017 o 00:44, Ondrej Zajicek pisze:

On Fri, Dec 15, 2017 at 03:42:03PM +, Rhys Williams wrote:

Hey,

I am also seeing this memory leak on Ubuntu 16.04, kernel 4.10.0-36.

The memory leak seems to occur when the scan timer lapses on the kernel 
protocol leaking about the full tables worth of memory each time until 
eventually bird is oom killed.

This patch should fix it.






smime.p7s
Description: Kryptograficzna sygnatura S/MIME

Re: Bird 2.0.0 memory leak

[Ondrej Zajicek]

On Fri, Dec 15, 2017 at 03:42:03PM +, Rhys Williams wrote:
> Hey,
> 
> I am also seeing this memory leak on Ubuntu 16.04, kernel 4.10.0-36.
> 
> The memory leak seems to occur when the scan timer lapses on the kernel 
> protocol leaking about the full tables worth of memory each time until 
> eventually bird is oom killed.

This patch should fix it.


-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."
diff --git a/sysdep/linux/netlink.c b/sysdep/linux/netlink.c
index 76ffe78b..790e7b23 100644
--- a/sysdep/linux/netlink.c
+++ b/sysdep/linux/netlink.c
@@ -1688,14 +1688,6 @@ nl_parse_route(struct nl_parse_state *s, struct nlmsghdr *h)
 ra->nh.labels = labels;
 #endif
 
-  rte *e = rte_get_temp(ra);
-  e->net = net;
-  e->u.krt.src = src;
-  e->u.krt.proto = i->rtm_protocol;
-  e->u.krt.seen = 0;
-  e->u.krt.best = 0;
-  e->u.krt.metric = 0;
-
   if (i->rtm_scope != def_scope)
 {
   ea_list *ea = lp_alloc(s->pool, sizeof(ea_list) + sizeof(eattr));
@@ -1709,9 +1701,6 @@ nl_parse_route(struct nl_parse_state *s, struct nlmsghdr *h)
   ea->attrs[0].u.data = i->rtm_scope;
 }
 
-  if (a[RTA_PRIORITY])
-e->u.krt.metric = rta_get_u32(a[RTA_PRIORITY]);
-
   if (a[RTA_PREFSRC])
 {
   ip_addr ps = rta_get_ipa(a[RTA_PREFSRC]);


signature.asc
Description: PGP signature

Re: (unfortunately) more BIRD-2.0.0 woes ...

[Ondrej Zajicek]

On Fri, Dec 15, 2017 at 07:14:36PM +0100, Ondrej Zajicek wrote:
> On Fri, Dec 15, 2017 at 05:59:09PM +0100, Clemens Schrimpe wrote:
> > I forgot:
> > 
> > 4. „dest = RTD_BLACKHOLE“ does appear to no longer be functioning as well … 
> > ;-(
> 
> Works for me. How and where it does not work?

Seems like it does not work when the original route has recursive next
hop, which in your case caused route oscillations and 100% cpu.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."


signature.asc
Description: PGP signature

Re: (unfortunately) more BIRD-2.0.0 woes ...

[Ondrej Zajicek]

On Fri, Dec 15, 2017 at 05:59:09PM +0100, Clemens Schrimpe wrote:
> I forgot:
> 
> 4. „dest = RTD_BLACKHOLE“ does appear to no longer be functioning as well … 
> ;-(

Works for me. How and where it does not work?

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."


signature.asc
Description: PGP signature

Re: Bird 2.0.0 memory leak

[Ondrej Zajicek]

On Fri, Dec 15, 2017 at 03:42:03PM +, Rhys Williams wrote:
> Hey,
> 
> I am also seeing this memory leak on Ubuntu 16.04, kernel 4.10.0-36.
> 
> The memory leak seems to occur when the scan timer lapses on the kernel 
> protocol leaking about the full tables worth of memory each time until 
> eventually bird is oom killed. 

Thanks, that is important hint. Will check that.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."


signature.asc
Description: PGP signature

Re: (unfortunately) more BIRD-2.0.0 woes ...

[Ondrej Zajicek]

On Fri, Dec 15, 2017 at 05:56:09PM +0100, Clemens Schrimpe wrote:
> Hello again -
> 
> a couple of days ago I converted one of my test route-server setups from 
> „dual BIRD-1.6.3“ to „single BIRD-2.0.0“ by simply „merging“ bird6.conf and 
> bird.conf into a single (unified) new bird.conf - also observing the 
> guidelines laid out in 
> https://gitlab.labs.nic.cz/labs/bird/wikis/transition-notes-to-bird-2 
>  . 
> 
> The results of this simple approach are between weird and stunning:
> 
> I now have a bird process which is consuming 100% of one CPU. It is
> still talking to me through birdc, but is very sluggish, i.e. bringing up
> peers or tearing them down takes a long time - you even see states, like
> „flushing“ in „show proto all XXX“ for quite some time, which you
> normally don’t even notice, because they are usually very transitive :-)
>
> I haven’t gotten deeper into the problem, yet, but there appears to be a loop 
> involving the affected peer’s Export Updates:
> 
> Route change stats: received   rejected   filteredignored   
> accepted
>   Import updates: 670962  0  0  0 
> 670962
>   Import withdraws:  216  0---  0
> 216
>   Export updates:  317034802  159525749  157509053--- 
>  0
>   Export withdraws:  468--------- 
>  0
> 
> (those numbers are increasing fast)
> 
> Some of my peers were/are already MP-BGP capable and so some IPv6 peers
> now came up with „Hey, you speak IPv4 too? Cool! Here are some 700.000
> routes for your, my friend!“, which is totally explicable, yet it
> surprised my a teeny bit, anyway :-)
> 
> It should be made clear in the documentation, that BGP protocol
> definitions without explicit address-family → „channel“ definitions my
> eventually (see below) ending up MP-BGP sessions with the above effect!?

Hi

This should not happen. In BGP, only explicitly configured channels are
created and announced. Perhaps you have both channels configured in the
template? That would be expected behavior, there is no difference between
channel configured in template or in protocol. It is valid use case to
have all channels configured in template and then just 'remote' in
protocol.

> Most surprising: My multiple-inheritance approach, using multiple,
> chained template definitions for my BGP „protocols“/sessions/peers broke
> in the weirdest way: I now have multiple IPv6 and IPv4 channels on some
> of my protocols with different parameterizations:
> ...
> (it seems the ipv6/ipv4 channel definitions within the templates each
> create their own instances instead of inheriting/enhancing (from) the
> ones upstream???)

This are two issues:

1) Nested options from templates are not merged, just concatenated.
Templates always behaved this way but it is true that with advent
of channel config blocks this make them less useful and change of
the behavior is perhaps necessary. At least i will add a note to
the guidelines.

2) Config file containing BGP with multiple channels of the same type
(either directly or through templates) is not rejected. This was already
discussed and will be fixed soon:

http://trubka.network.cz/pipermail/bird-users/2017-December/011704.html


> @Ondrej, et.al., is this even a „supported mode of operation“ → multiple IPv6 
> and IPv4 channels within one protocol?

No

> So this still somehow works, yet still with the 100% CPU consumption.
> 
> @bird-devs: How would you like to address above issues? Yes, I can
> probably clean up the config file, which would probably even make the
> 100%-issue go away. But imho it shouldn’t be possible in the first place
> to create configs like these without BIRD at least kicking and screaming
> loudly about it.

Yes, i will fix the #2 so that such config files are rejected, and perhaps 
template-channel merging, but it is not a simple bug such that i will send
a patch immediately.

It would be a good idea if you could try to clean up the config file
to eliminate duplicit channels. I guess that will also fix the 100%
issue. I will send you a ssh-pubkey.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."


signature.asc
Description: PGP signature

Re: (unfortunately) more BIRD-2.0.0 woes ...

[Clemens Schrimpe]

I forgot:

4. „dest = RTD_BLACKHOLE“ does appear to no longer be functioning as well … ;-(

Clemens

Re: birdc 2.0.0 crashes

[Ondrej Zajicek]

On Fri, Dec 15, 2017 at 04:16:31PM +0100, Clemens Schrimpe wrote:
> > Could you try to build bird/birdc (without any patches i sent before) using:
> > 
> >  make CLIENT_LIBS="-lreadline -ltinfo"
> > 
> > ?
> 
> And indeed, this fixes it and persistent history still works! 
> 
> PS: for your records (again) - this machine run Ubuntu 16.04.3 LTS in a
> (mostly) vanilla fashion. I just installed libreadline6-dbg to help with
> this case. So the same problem/solution *should* apply on other machines
> as well!?

Yes, the bad linking is on most other systems, it just does not mainfest
on them.

Thanks to all for debugging the issue.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."


signature.asc
Description: PGP signature

Re: Bird 2.0.0 memory leak

[Rhys Williams]

Hey,

I am also seeing this memory leak on Ubuntu 16.04, kernel 4.10.0-36.

The memory leak seems to occur when the scan timer lapses on the kernel 
protocol leaking about the full tables worth of memory each time until 
eventually bird is oom killed. 

Regards,

Rhys Williams

December 15, 2017 2:55 PM, "Miłosz Oller"  wrote:

> Hi
> 
> After install bird 2.0.0 from sources, bird has memory leak. After 
> running it c.a. 15min daemon used ~24G of RAM.
> 
> Compile options: ./configure --enable-client --enable-libssh
> 
> Debian 9, kernel 4.9.69
> 
> -- 
> Miłosz

Re: birdc 2.0.0 crashes

[Clemens Schrimpe]

> Could you try to build bird/birdc (without any patches i sent before) using:
> 
>  make CLIENT_LIBS="-lreadline -ltinfo"
> 
> ?

And indeed, this fixes it and persistent history still works! 

\o/

Clemens

PS: for your records (again) - this machine run Ubuntu 16.04.3 LTS in a 
(mostly) vanilla fashion. I just installed libreadline6-dbg to help with this 
case. So the same problem/solution *should* apply on other machines as well!?
 

Re: Bird 2.0.0 memory leak

[Jan Maria Matejka]

Hi!

On 12/15/2017 02:45 PM, Miłosz Oller wrote:
> Hi
> 
> After install bird 2.0.0 from sources, bird has memory leak. After running it 
> c.a. 15min daemon used ~24G of RAM.
> 
> Compile options: ./configure --enable-client --enable-libssh
> 
> Debian 9, kernel 4.9.69

Could you please send in the config file you are running Bird with?

Thanks
JMM

Bird 2.0.0 memory leak

[Miłosz Oller]

Hi

After install bird 2.0.0 from sources, bird has memory leak. After 
running it c.a. 15min daemon used ~24G of RAM.


Compile options: ./configure --enable-client --enable-libssh

Debian 9, kernel 4.9.69

--
Miłosz




smime.p7s
Description: Kryptograficzna sygnatura S/MIME

Re: Bird Installation Guide

[Pardeep Sharma]

Hi Mo,

You are right, I have doubt whether bird is running or not because I run in
linux "yum install bird" command and "bird.conf" file generated in /etc in
root user. In my knowledge its may the installation of BIRD but in real it
doesn't seem because I following links shared by you but there installation
is based on 'Unix' OS not Linux.

If you have info, how to install BIRD in linux, please share with me
because I searched a lot in google BIRD installation in Linux but didn't
get. After running bird -c its shows "no such file or directory"

Thanks a lot


On 15 December 2017 at 17:05, Moyaze Shivji  wrote:

>
> On 15 Dec 2017, at 11:26, Pardeep Sharma 
> wrote:
>
> Hi Moyaze/BIRD EXPERTS,
>
> Thanks a lot Moyaze.. providing support..
> Appreciate your help Moyaze..
>
> when I am running "birdc" in linux its gives error as attached. Please see
> the attachment.. I had gone some docs but didn;t resolution. Please guide
> me for next..
>
>
> Are you sure BIRD is running on your system ?
>
> try doing
>
> sudo bird -c  “/”
>
> then do
>
> sudo birdc.
>
> Mo
>
>
>


-- 
Thanks & Regards
Pardeep

Re: Bird Installation Guide

[Moyaze Shivji]

> On 15 Dec 2017, at 11:26, Pardeep Sharma  wrote:
> 
> Hi Moyaze/BIRD EXPERTS,
> 
> Thanks a lot Moyaze.. providing support.. 
> Appreciate your help Moyaze..
> 
> when I am running "birdc" in linux its gives error as attached. Please see 
> the attachment.. I had gone some docs but didn;t resolution. Please guide me 
> for next..
> 


Are you sure BIRD is running on your system ?

try doing 

sudo bird -c  “/”

then do

sudo birdc.

Mo

Re: Bird Installation Guide

[Moyaze Shivji]

> On 15 Dec 2017, at 10:02, Pardeep Sharma  wrote:
> 
> Hi Moyaze,
> 
> Thanks for your best support here...
> 
> Can u please guide us how to get BGP access in BIRD via HTTPs. I have just 
> installed BIRD in Linux machine because rightnow I am able to see only 
> bird.conf in /etc/bird.conf ini linux OS. Is there a way to access BIRD as 
> similar to quagga server or normal router ?
> 

To get access to the CLI do

sudo birdc 

from there you can enter commands.

This looking-glass may help in getting some kind of web access.

https://gitlab.labs.nic.cz/labs/ulg/

There are other looking-glass’s . Just google for them.

Mo

Re: Bird Installation Guide

[Pardeep Sharma]

Hi Moyaze,

Thanks for your best support here...

Can u please guide us how to get BGP access in BIRD via HTTPs. I have just
installed BIRD in Linux machine because rightnow I am able to see only
bird.conf in /etc/bird.conf ini linux OS. Is there a way to access BIRD as
similar to quagga server or normal router ?

Thanks a lot

On 15 December 2017 at 15:19, Moyaze Shivji  wrote:

>
> On 15 Dec 2017, at 05:57, Pardeep Sharma 
> wrote:
>
> Hi Moyaze,
>
> Thanks a lot for your help.
>
> The  above links are very helpful and I have installed BIRD in my linux
> 7.0.4. But now, I am having issue, is that, how I can verify the BIRD BGP
> routing table and its routes learning from other PEERS.
>
>
>
> This mate help, its shows what cisco commands are equivalent in BIRD.
>
> https://gitlab.labs.nic.cz/labs/bird/wikis/Command_interface_examples
>
> typical commands are:
>
> show route ~ show ip route
> show protocols ~ sh ip bgp summary
> show protocols all  ~ sh ip bgp neighbor
>
>
> hope this helps.
>
> Mo
>
>
>


-- 
Thanks & Regards
Pardeep

Re: Bird Installation Guide

[Moyaze Shivji]

> On 15 Dec 2017, at 05:57, Pardeep Sharma  wrote:
> 
> Hi Moyaze,
> 
> Thanks a lot for your help.
> 
> The  above links are very helpful and I have installed BIRD in my linux 
> 7.0.4. But now, I am having issue, is that, how I can verify the BIRD BGP 
> routing table and its routes learning from other PEERS. 
> 


This mate help, its shows what cisco commands are equivalent in BIRD.

https://gitlab.labs.nic.cz/labs/bird/wikis/Command_interface_examples

typical commands are:

show route ~ show ip route
show protocols ~ sh ip bgp summary
show protocols all  ~ sh ip bgp neighbor


hope this helps.

Mo