Package: dpkg Version: 1.21.9 Severity: normal X-Debbugs-Cc: m...@debian.org, debian-wb-t...@lists.debian.org
(Setting initial severity as "normal" because it affects packages built in buildd machines). Hi, The origins of this bug report are because there are sometimes problems building packages in buildds, the compression phase is very slow and sometimes the build is aborted due to inactivity: E: Build killed with signal TERM after 300 minutes of inactivity [1] After some investigation by aurel32 and myself, this was traced back to the commit f8d254943051e085040367d689048c00f31514c3 [2], in which the calculation of the memory that can be used, to determine the number of threads to use, was changed from half of the physical mem to be based on the memory available. For example in buildds with not-so-large amount of RAM, using part of it for tempfs (which is how buildds are set-up), the reported available memory maybe is not very large. For example it could be MemAvailable=2GB for 16GB of physical RAM in the machine. In the dpkg code, for the purposes of the calculation of how much memory can be used, the result appears to be to be half of the MemAvailable [3], so only 1GB. According to the tables in xz man page, for compression the algorithm can use 674MB. So as a result, dpkg-deb uses single-threaded compression, even if it could easily use 2-3 threads and still use only RAM; or use the full number of threads in the machine (e.g. 4 or 8) even if it means swapping out some of the content of tempfs -- which is not a problem in most cases for buildds, specially if using fast disks. As a result of all this, it is taking upwards of 600 mins of CPU time to compress recent linux-image-*-dbg packages in the buildds of riscv64 architecture at the moment, so when using 2 threads of less, it's guaranteed to be aborted. But this also affects other arches and other packages in other ways, at least by making the build needlessly slow in many cases. It's not very clear what could be the solution for this, as the default could be OK for desktops and many machines in which there's plenty of available RAM, but this is not the case of all of our buildds. It might not be possible to detect which is the best from within dpkg. Maybe it could be through a new variable DPKG_DEB_THREADS (similar to the recently added DPKG_DEB_MAX_THREADS) in which the buildds explicitly set the amount of threads that they want. PS: Bug #501456 [4] is possibly a duplicate, or at least the intersection is largish :) [1] https://buildd.debian.org/status/fetch.php?pkg=linux&arch=riscv64&ver=6.1%7Erc3-1%7Eexp1&stamp=1667646869&raw=0 [2] https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/?id=f8d254943051e085040367d689048c00f31514c3 [3] At least that's what I got in my tests as value for mt_memlimit, half of the MemAvailable at the time, even if I expected the full amount of MemAvailable when quickly reading the code. But I didn't pay much attention because this detail is not very important in principle: MemAvailable could be very low if tempfs is using more than the physical RAM, it doesn't matter if it's full or half in many cases with low memory. [4] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=501456 Thanks and cheers. -- Manuel A. Fernandez Montecelo <m...@debian.org>