Helmut Grohne:
Hi Guillem and other developers,I am one of those who builds a lot of different packages with different requirements and found that picking a good parallel=... value in DEB_BUILD_OPTIONS is hard. Go too low and your build takes very long. Go too high and you swap until the OOM killer terminates your build. (Usage of choom recommended in any case.) [...] I think this demonstrates that we probably have something between 10 and 50 packages in unstable that would benefit from a generic parallelism limit based on available RAM. Do others agree that this is a problem worth solving in a more general way? For one thing, I propose extending debhelper to provide --min-ram-per-parallel-core as that seems to be the most common way to do it. I've proposed https://salsa.debian.org/debian/debhelper/-/merge_requests/128 to this end. Unfortunately, a the affected packages tend to not just be big, but also so special that they cannot use dh_auto_*. As a result, I also looked at another layer to support this and found /usr/share/dpkg/buildopts.mk, which sets DEB_BUILD_OPTION_PARALLEL by parsing DEB_BUILD_OPTIONS. How about extending this file with a mechanism to reduce parallelity? I am attaching a possible extension to it to this mail to see what you think. Guillem, is that something you consider including in dpkg?
My suggestion would be to have `dpkg` have a "RAM restrained" parallelization limit next to the default one. This is similar to how the `debhelper` one works (the option only applies to the dh_auto_ steps). This might be what you are proposing here (was unsure).
Generally, the RAM limit only applies to the upstream build side of things and this can be relevant for some packages. As an example, `firefox-esr` builds 20+ packages, so there is value in having "post processing" (dh_install ... dh_builddeb) happening with a higher degree of parallelization than the upstream build part to keep the build time as low as possible.
I think 2 degrees of parallelization limits would be sufficient for most cases (at least from a cost/benefit PoV).
Are there other layers that could reasonably be used to implement a more general form of parallelism limiting based on system RAM? Ideally, we'd consolidate these implementations into fewer places. [...] Helmut
We do have some custom implementations of debhelper build systems around in the archive[1], that in theory could do with this (though they are probably not worth hunting out - more of "update if they become a problem"). Then there is `debputy`, but I can have a look at that later after I have reviewed the patch for `debhelper`.
Best regards, Niels[1] Any `dh-sequence-X` sequences that replace `dh_auto_*` commands would likely fall into this category.
OpenPGP_signature.asc
Description: OpenPGP digital signature
