Re: Bird 1.5 and 1.6 memory leak

[Ondrej Zajicek]

On Wed, Apr 18, 2018 at 09:57:16AM +0200, Willy Goiffon wrote:
> Le 2018-04-17 14:27, Alexander Demenshin a écrit :
> > On 2018-04-17 13:31, wi...@mailoo.org wrote:
> > 
> > > As I said, the mem leak does not apppear in the `birdc show memory`
> > > command.
> > > It is shown at the OS level, and is shown via the `free` command.
> > 
> > What do you mean by "OS level"? Which exactly level? What is the actual
> > process
> > size as reported by ps ("ps axuw|grep bird")?
> 
> By "OS Level" I mean the memory used as reported by the kernel from
> /proc/meminfo.

Hi

Just curios, which field in /proc/meminfo (or /proc/slabinfo)
is increasing?

> > If ps shows size similar to what you see in birdc "show mem", then the
> > leak
> > is definitely not in bird (though it may be related) - it is in kernel
> > or some other process.
> 
> It might not be in bird, but it is definitely triggered by it.

That is interesting, one could imagine that difference between running
BIRD and just using static routes would be that periodic scans could
trigger some kernel bug, but that would cause linear increase independent
of network traffic.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."

Re: Bird 1.5 and 1.6 memory leak

[Willy Goiffon]

Le 2018-04-17 14:27, Alexander Demenshin a écrit :

On 2018-04-17 13:31, wi...@mailoo.org wrote:

As I said, the mem leak does not apppear in the `birdc show memory` 
command.

It is shown at the OS level, and is shown via the `free` command.


What do you mean by "OS level"? Which exactly level? What is the actual 
process

size as reported by ps ("ps axuw|grep bird")?


By "OS Level" I mean the memory used as reported by the kernel from 
/proc/meminfo.

The process size is rather standard:

bird 1737 0.0 0.0 15540 2216 ? Ss Apr17 0:01 /usr/sbin/bird -f -u 
bird -g bird


If I sum up the size of all processes, I don't even reach 1Gib. But the 
memory used

reported by the kernel increases gradually until it reaches OOM.

If ps shows size similar to what you see in birdc "show mem", then the 
leak

is definitely not in bird (though it may be related) - it is in kernel
or some other process.


It might not be in bird, but it is definitely triggered by it.

Thanks
--
Willy

Re: Bird 1.5 and 1.6 memory leak

[Willy Goiffon]

if it occurs also on 1.6.4, could you please add also "birdcl show
route count" output and some logs with debug all switched on?


I had to reinstall version 1.5.0, to remain in the ubuntu default PPA.
The route count is ridiculously low though:

$ birdc show route count
BIRD 1.5.0 ready.
11 of 11 routes for 11 networks


How does it get leaked, do you reconfigure many times? Does the memory
consumption increase gradually or by larger amounts?


It seems to leak at the kernel level, as the process memory doesn't 
increase.

I'm not doing any reconfiguration. It increases gradually and is closely
tied to the traffic crossing the router.
If I stop any traffic from passing (eg, using iptables), the memleak 
stops.


As I said, I also don't have this leak if I leave bird down and 
configure

the routes statically.

Re: Bird 1.5 and 1.6 memory leak

[Alexander Demenshin]

On 2018-04-17 13:31, wi...@mailoo.org wrote:

As I said, the mem leak does not apppear in the `birdc show memory` 
command.

It is shown at the OS level, and is shown via the `free` command.


What do you mean by "OS level"? Which exactly level? What is the actual 
process

size as reported by ps ("ps axuw|grep bird")?

If ps shows size similar to what you see in birdc "show mem", then the 
leak

is definitely not in bird (though it may be related) - it is in kernel
or some other process.

--
Best regards,
Alexander.

Re: Bird 1.5 and 1.6 memory leak

[willy]

> debian stretch, bird v 1.6.3-2 is rock solid
> 
> FWIW i saw zero leaks or issues on debian wheezy, jessie, devuan jessie 
> either.
> 
> on multiple rrs taking several million routes I am at:
> 
> birdc
> BIRD 1.6.3 ready.
> bird> show mem
> BIRD memory usage
> Routing tables:126 MB
> Route attributes:   93 MB
> ROA tables:112  B
> Protocols: 823 kB
> Total: 219 MB

As I said, the mem leak does not apppear in the `birdc show memory` command.
It is shown at the OS level, and is shown via the `free` command.
The leak was also present with v1.6.3  on ubuntu though.
Note that I don't have that much routes (~20), so the leak is not related to 
that IMO.

> alpine linux also works very well, no issues taking multiple full tables 
> on core/edge as well:
> 
> (host: core1)  (user: root)  (time= 11:23:53)
>   > ~ # birdc
> BIRD 1.6.0 ready.
> bird> show mem
> BIRD memory usage
> Routing tables:182 MB
> Route attributes:  156 MB
> ROA tables:192  B
> Protocols: 686 kB
> Total: 339 MB
> bird>

Unfortunately, switching to alpine or debian is not an option in my case.

Thank you for the input!

Re: Bird 1.5 and 1.6 memory leak

[Jan Maria Matějka]

Hello,
if it occurs also on 1.6.4, could you please add also "birdcl show route count" 
output and some logs with debug all switched on?

How does it get leaked, do you reconfigure many times? Does the memory 
consumption increase gradually or by larger amounts?

Thanks
Maria

Quidquid latine dictum sit, altum videtur. 

-Original Message-
From: Willy Goiffon 
To: bird-users@network.cz
Sent: Fri, 13 Apr 2018 6:21 PM
Subject: Bird 1.5 and 1.6 memory leak

Hello,

I installed different version of bird on Ubuntu 16.04 (namely, 1.5.0, 
1.6.3, 1.6.4).
After leaving the daemon running for ~1 hour, the memory used by the
host is over 8Gib. However, when running `birdc show memory`, I get
the following:

 BIRD memory usage
 Routing tables: 21 kB
 Route attributes:   11 kB
 ROA tables:192  B
 Protocols:  22 kB
 Total: 102 kB

My config is pretty basic (edited, see attached) as we have 2 BGP
protocols using 2 ASN (one public, one private), each exporting the
routes learnt from the other.

The leak seems to be related to the traffic going through the server,
as the memory of the host remains still when there is no traffic
going on. It is triggered by bird though, because I don't have the
leak when the routes are configured statically and bird is shut down.

The weird thing about this is that the memory is not used by the bird
process itself. It's shown as used memory by the host, but no process
is actually using this memory.
I know that it's leaking because of bird though because it's the only
process installed and running on this host.

Has anyone ever encountered such a behavior, or can see an issue in
my configuration?

Thanks in advance for your help
-- 
Willy

Re: Bird 1.5 and 1.6 memory leak

[C. Jon Larsen]

debian stretch, bird v 1.6.3-2 is rock solid

FWIW i saw zero leaks or issues on debian wheezy, jessie, devuan jessie 
either.


on multiple rrs taking several million routes I am at:

birdc
BIRD 1.6.3 ready.
bird> show mem
BIRD memory usage
Routing tables:126 MB
Route attributes:   93 MB
ROA tables:112  B
Protocols: 823 kB
Total: 219 MB

alpine linux also works very well, no issues taking multiple full tables 
on core/edge as well:


(host: core1)  (user: root)  (time= 11:23:53)
 > ~ # birdc
BIRD 1.6.0 ready.
bird> show mem
BIRD memory usage
Routing tables:182 MB
Route attributes:  156 MB
ROA tables:192  B
Protocols: 686 kB
Total: 339 MB
bird>





On Fri, 13 Apr 2018, Willy Goiffon wrote:


Hello,

I installed different version of bird on Ubuntu 16.04 (namely, 1.5.0, 1.6.3, 
1.6.4).

After leaving the daemon running for ~1 hour, the memory used by the
host is over 8Gib. However, when running `birdc show memory`, I get
the following:

   BIRD memory usage
   Routing tables: 21 kB
   Route attributes:   11 kB
   ROA tables:192  B
   Protocols:  22 kB
   Total: 102 kB

My config is pretty basic (edited, see attached) as we have 2 BGP
protocols using 2 ASN (one public, one private), each exporting the
routes learnt from the other.

The leak seems to be related to the traffic going through the server,
as the memory of the host remains still when there is no traffic
going on. It is triggered by bird though, because I don't have the
leak when the routes are configured statically and bird is shut down.

The weird thing about this is that the memory is not used by the bird
process itself. It's shown as used memory by the host, but no process
is actually using this memory.
I know that it's leaking because of bird though because it's the only
process installed and running on this host.

Has anyone ever encountered such a behavior, or can see an issue in
my configuration?

Thanks in advance for your help
--
Willy