On Fri, 10 Jul 2026 16:10:09 +0200 Maria Matejka <[email protected]> wrote:
> Hi Mariusz,
>
> On Fri, Jul 10, 2026 at 03:19:03PM +0200, Mariusz Gronczewski wrote:
>
> > I've been observing some memory leak (slowly gets to ~30GB across 18
> > hours then dies to oomkiller, ~500kB/s growth), how can I debug it
> > furter ?
>
> That is not an easy task, and I'm a little worried about your actual
> memory footprint. This is what I get with one uplink:
>
> BIRD 3.3.1 ready.
> bird> show route count
> 1047039 of 1047039 routes for 1047039 networks in table master4
> 245208 of 245208 routes for 245208 networks in table master6
> Total: 1292247 of 1292247 routes for 1292247 networks in 2 tables
> bird> show mem
> BIRD memory usage
> Effective Overhead
> Routing tables: 191.7 MB 12.8 MB
> Route attributes: 115.3 MB 70.2 MB
> Protocols: 371.2 kB 13.6 kB
> Current config: 17.0 kB 880.0 B
> Standby memory: 0.0 B 33.2 MB
> Total: 307.5 MB 116.3 MB
>
> Active pages: 308.2 MB
> Kept free pages: 33.2 MB
> Cold free pages: 129.0 MB
> Hot page cache depleted while in RCU: 2482
we just have 2 peers with full tables so yeah, route count doesn't
really match actually used memory
> > # birdc
> > BIRD 3.3.1 ready.
> > bird> show memory
> > BIRD memory usage
> > Effective Overhead
> > Routing tables: 6059.3 MB 564.4 MB
> > Route attributes: 9510.1 MB 565.8 MB
> > Protocols: 22.3 MB 82.9 kB
> > Current config: 60.8 kB 1232.0 B
> > Standby memory: 0.0 B 32.7 MB
> > Total: 15.2 GB 1163.0 MB
> >
> > Active pages: 16.2 GB
> > Kept free pages: 32.7 MB
> > Cold free pages: 194.2 MB
> > Hot page cache depleted while in RCU: 66929
> >
> > bird> show route count
> > 2098594 of 2098594 routes for 1051498 networks in table master4
> > 0 of 0 routes for 0 networks in table master6
>
> This consumption is in itself very fishy. My first look would be into
> your filters because 10G in route attributes is probably terribly
> wrong with just 2M routes, and there must be _some_ reason for that,
> probably impossible deduplication.
> Another look would be what happens when you slightly change your
> import filter (e.g. by adding yet another community) and see whether
> the reported memory footprint raises significantly. Also, it may be
> handy to set
>
> debug protocols { states };
> debug channels { routes, states };
> debug tables { routes, states };
>
> and see what is happening over time, but as it is slow, it may yield
> nothing.
Oh it isn't slow at all, in my case for some reason there is ton of
0.0.0.0/0 route updates
like, kernel get good 2 orders of magnitude more updates than any of
the BGP peers produces
Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and
only generated in static and then distributed to OSPF:
protocol static generated {
ipv4 {
table generated_t;
};
check link;
igp table master4;
route 193.200.227.0/24 blackhole;
route 194.50.132.0/22 blackhole;
route 0.0.0.0/0 recursive 8.8.8.8 {
preference=500;
};
route 0.0.0.0/0 recursive 1.1.1.1 {
preference=290;
};
route 0.0.0.0/0 recursive 13.96.0.1 {
preference=280;
};
route 0.0.0.0/0 recursive 142.250.0.1 {
preference=270;
};
route 0.0.0.0/0 recursive 18.244.0.1 {
preference=260;
};
route 0.0.0.0/0 unreachable {
preference=1;
};
};
That was my poor attempt at trying to generate one only on router that actually
sees internet (and not just have peer up), I wanted to steer traffic to router
that have "working" ISP, but I'm not sure why it would generate such flapping
(and either way shouldnt really leak memory)
I've attached log and full config
> > [...]
> > learn all;
>
> This may play some role but I don't believe that as the implementation
> of the kernel protocol almost didn't change between 3.1 and 3.3.
>
> > It only started happening once router was enabled (before it was
> > peered but did not push the routes to kernel table), tho it
> > coincided with upgrade from 3.1 to 3.3.1 as I hit different bug
> > there .
> >
> > I have also tried to disable "export/import table", "import keep
> > filtered on" and "allow as sets" just to narrow it down but appears
> > there is no change
>
> I would expect that with export/import table off, the memory
> consumption would go away.
No change after disabling, still grows
> By any chance, how much are you calling `birdc`? We got reported some
> memory leaks regarding that.
saw that thread, I've tried disabling montioring (we do have prometheus
exporter exporting it quite often) but no change
>
> Also, I expect that your OSPF is not producing way too many routes,
> but if you could elaborate on that number, it may be a clue.
around 100 or so so probably not
> Also, `import keep filtered` in OSPF is not needed, there is no
> benefit from setting it.
good to know, thanks
>
> Thanks for the report!
>
--
Mariusz Gronczewski (XANi) <[email protected]>
GnuPG: 0xEA8ACE64
https://devrandom.eu
bird.log.gz
Description: application/gzip
config.tar.gz
Description: application/gzip
