>> A question I had: is it really worth 2k of complicated kernel code to >> *maybe sometimes* save <10Mb of memory? > > There are many reasons this is worth it. The main ones: > > 1. This did not start from a number we picked, but from a real use > case: users who run large fleets of small instances, 1 GiB of > memory or less, with their workloads sized to fit. We are not able > to disclose more detail, but for them 5.4 MB on every instance is > real money, and it can be exactly what pushes a workload over its > memory budget and onto the next instance size. That is why we took > this on, even though we knew it would not be a small change. > > 2. Loading code and data only when a system needs them is what kernel > modules exist for. And most of them take well under ~1 MB once > loaded (nf_conntrack, overlay, vfat); even big ones like ext4, kvm > and btrfs stay around 1-2.5 MB. The vmlinux BTF is 5.4 MB. > > 3. We are not alone in trying to keep BTF out of memory until it is > needed. The inline BTF work [1] plans to deliver its data, which > is even larger, through a module too. The .BTF.link record and the > resolve_btfids option that serve both are already part of this > series (patch 10), so this is not machinery for the vmlinux BTF > alone.
The other consideration that embedded folks have raised is disk footprint; apparently by moving vmlinux .BTF to a module that is a win for them due to the way embedded systems are partitioned (vmlinux image on a different partition from modules). And for them a lower memory footprint is a win too. So from my perspective it is worth doing, especially if the only other option is to switch BTF off and lose most BPF functionality. Between small VMs and embedded systems that could amount to a lot of systems. However it may make sense to approach via a multi-phase delivery. The inline stuff is pretty close, so if I can land that with the resolve_btfids support for --btf_link that reduces the footprint of this work somewhat. Then perhaps it might make sense to split into a prerequisite series that does the prep work for module-delivered BTF (roughly patches 1-5), avoiding issues around module load by reorganizing when we access vmlinux BTF. Then the remainder - stuff which is tied more closely to module vmlinux BTF delivery - would be a smaller series (I'm planning on sending out the inline kbuild stuff later today all going well). Others may have different suggestions, but that may make the work easier to land; the multi-series approach definitely helped with the inline work FWIW. Alan
