On Fri, 10 Jul 2026 16:10:09 +0200 Maria Matejka <[email protected]> wrote:
> Hi Mariusz,
>
> On Fri, Jul 10, 2026 at 03:19:03PM +0200, Mariusz Gronczewski wrote:
>
> > I've been observing some memory leak (slowly gets to ~30GB across 18
> > hours then dies to oomkiller, ~500kB/s growth), how can I debug it
> > furter ?
>
> That is not an easy task, and I'm a little worried about your actual
> memory footprint. This is what I get with one uplink:
>
> BIRD 3.3.1 ready.
> bird> show route count
> 1047039 of 1047039 routes for 1047039 networks in table master4
> 245208 of 245208 routes for 245208 networks in table master6
> Total: 1292247 of 1292247 routes for 1292247 networks in 2 tables
> bird> show mem
> BIRD memory usage
> Effective Overhead
> Routing tables: 191.7 MB 12.8 MB
> Route attributes: 115.3 MB 70.2 MB
> Protocols: 371.2 kB 13.6 kB
> Current config: 17.0 kB 880.0 B
> Standby memory: 0.0 B 33.2 MB
> Total: 307.5 MB 116.3 MB
>
> Active pages: 308.2 MB
> Kept free pages: 33.2 MB
> Cold free pages: 129.0 MB
> Hot page cache depleted while in RCU: 2482
>
> > # birdc
> > BIRD 3.3.1 ready.
> > bird> show memory
> > BIRD memory usage
> > Effective Overhead
> > Routing tables: 6059.3 MB 564.4 MB
> > Route attributes: 9510.1 MB 565.8 MB
> > Protocols: 22.3 MB 82.9 kB
> > Current config: 60.8 kB 1232.0 B
> > Standby memory: 0.0 B 32.7 MB
> > Total: 15.2 GB 1163.0 MB
> >
> > Active pages: 16.2 GB
> > Kept free pages: 32.7 MB
> > Cold free pages: 194.2 MB
> > Hot page cache depleted while in RCU: 66929
> >
> > bird> show route count
> > 2098594 of 2098594 routes for 1051498 networks in table master4
> > 0 of 0 routes for 0 networks in table master6
>
> This consumption is in itself very fishy. My first look would be into
> your filters because 10G in route attributes is probably terribly
> wrong with just 2M routes, and there must be _some_ reason for that,
> probably impossible deduplication.
>
> Another look would be what happens when you slightly change your
> import filter (e.g. by adding yet another community) and see whether
> the reported memory footprint raises significantly. Also, it may be
> handy to set
>
> debug protocols { states };
> debug channels { routes, states };
> debug tables { routes, states };
>
> and see what is happening over time, but as it is slow, it may yield
> nothing.
>
oh no it's very fast, I think I have possibly found the problem
(attached log)
I noticed there is many changes in 0.0.0.0/0 route but I:
* don't import it from any peer
* export it only from OSPF
* is generated only as static route :
protocol static generated {
ipv4 {
table generated_t;
};
check link;
igp table master4;
route 193.200.227.0/24 blackhole;
route 194.50.132.0/22 blackhole;
route 0.0.0.0/0 recursive 8.8.8.8 {
preference=500;
};
route 0.0.0.0/0 recursive 1.1.1.1 {
preference=290;
};
route 0.0.0.0/0 recursive 13.96.0.1 {
preference=280;
};
route 0.0.0.0/0 recursive 142.250.0.1 {
preference=270;
};
route 0.0.0.0/0 recursive 18.244.0.1 {
preference=260;
};
route 0.0.0.0/0 unreachable {
preference=1;
};
};
This was poor attempt (I really wanted to avoid external scripting) at
generating a route preference based on whether a given router "sees"
wider internet (vs just sending traffic to peer that's).
But I don't understand why it causes so many updates
> > [...]
> > learn all;
>
> This may play some role but I don't believe that as the implementation
> of the kernel protocol almost didn't change between 3.1 and 3.3.
>
> > It only started happening once router was enabled (before it was
> > peered but did not push the routes to kernel table), tho it
> > coincided with upgrade from 3.1 to 3.3.1 as I hit different bug
> > there .
> >
> > I have also tried to disable "export/import table", "import keep
> > filtered on" and "allow as sets" just to narrow it down but appears
> > there is no change
>
> I would expect that with export/import table off, the memory
> consumption would go away.
nope
> By any chance, how much are you calling `birdc`? We got reported some
> memory leaks regarding that.
Yes, I saw that report, after stopping all of the tooling that does it
doesn't really change so I don't think it is related
> Also, I expect that your OSPF is not producing way too many routes,
> but if you could elaborate on that number, it may be a clue.
few hundred at most
> Also, `import keep filtered` in OSPF is not needed, there is no
> benefit from setting it.
>
> Thanks for the report!
>
--
Mariusz Gronczewski (XANi) <[email protected]>
GnuPG: 0xEA8ACE64
https://devrandom.eu
bird.log.gz
Description: application/gzip
config.tar.gz
Description: application/gzip
