[I was curious about what had large or huge install sizes --and how many for my example thresholds.]
On Mar 9, 2025, at 15:45, Mark Millard <[email protected]> wrote: > On Mar 9, 2025, at 12:34, Mark Millard <[email protected]> wrote: > >> On Mar 9, 2025, at 01:02, Mark Millard <[email protected]> wrote: >> >>> In looking for why the builders used notable tmpfs space >>> even when only one builder was left that was active, >>> I discovered that, for example, each builder ends up >>> with its own copy of /usr/local/poudriere/data/.m/*/*/usr/ >>> (and more) that does not end up being cleared out while >>> the builder is inactive. This looked to be a systematic >>> contribution to the tmpfs usage during times when various >>> builders are inactive. >>> >>> df -m /usr/local/poudriere/data/.m/main-aarch64-default/*/*/ | sed -e >>> 's@/[0-9][0-9]/@/*/@' | sort -k1,1 -k6,6 -k3,3 -k1,6 -u >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue 1114846 >>> 498917 526741 49% >>> /usr/local/poudriere/data/.m/main-aarch64-default/*/rescue >>> /usr/local/poudriere/data/packages/main-aarch64-default/.building 1114846 >>> 498917 526741 49% >>> /usr/local/poudriere/data/.m/main-aarch64-default/*/packages >>> /usr/local/poudriere/data/packages/main-aarch64-default/.building 1114846 >>> 498917 526741 49% >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/packages >>> /usr/local/poudriere/jails/main-aarch64/rescue 1114846 >>> 498917 526741 49% >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue >>> /usr/ports/distfiles 1114846 >>> 498917 526741 49% >>> /usr/local/poudriere/data/.m/main-aarch64-default/*/distfiles >>> /usr/ports/distfiles 1114846 >>> 498917 526741 49% >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/distfiles >>> Filesystem 1M-blocks >>> Used Avail Capacity Mounted on >>> devfs 0 >>> 0 0 0% >>> /usr/local/poudriere/data/.m/main-aarch64-default/*/dev >>> devfs 0 >>> 0 0 0% >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/dev >>> procfs 0 >>> 0 0 0% >>> /usr/local/poudriere/data/.m/main-aarch64-default/*/proc >>> procfs 0 >>> 0 0 0% >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/proc >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/01 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/02 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/03 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/04 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/05 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/06 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/07 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/08 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/09 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/10 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/11 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/12 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/13 >>> tmpfs 66539 >>> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/14 >>> tmpfs 66539 >>> 1140 65398 2% >>> /usr/local/poudriere/data/.m/main-aarch64-default/ref >>> >>> For the above, only ref/ and one other were active at the time. >>> Imagine having 32 builders or 128 builders or even more with >>> 1140 MiBytes for each inactive one. For the, above all the >>> builders actually reported: >>> >>> Inspecting . . .: determining shlib requirements >>> >>> for each package build it tried. No actual builds were done. >>> >>> [I'll note that the world directory trees for the booted >>> system and for the poudriere jail are from official >>> PkgBase builds that were installed. Also, the system >>> is UFS based, not ZFS based.] >>> >>> For reference (note the use of -x): >>> >>> # du -xsAm /usr/local/poudriere/data/.m/main-*-default/*/[uv][sa]r/ | sed >>> -e 's@/[0-9][0-9]/@/*/@' | sort -k1,2 -u >>> 376 /usr/local/poudriere/data/.m/main-aarch64-default/*/var/ >>> 376 /usr/local/poudriere/data/.m/main-aarch64-default/ref/var/ >>> 713 /usr/local/poudriere/data/.m/main-aarch64-default/*/usr/ >>> 713 /usr/local/poudriere/data/.m/main-aarch64-default/ref/usr/ >>> >>> So 1089 MiBytes or so of the 1140 MiBytes for each such tmpfs >>> is contributed overall by the combination of var/ and usr/ for >>> the example. >>> >>> # poudriere jail -l >>> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP >>> PATH >>> . . . >>> main-aarch64 15.0-CURRENT aarch64 pkgbase 2025-02-12 >>> 22:49:27 /usr/local/poudriere/jails/main-aarch64 >>> . . . >>> >>> There is more to look into here, in that, historically, larger >>> builders leave behind larger tmpfs usage until the next builder >>> reuse (if any). Having a few of lang/rust , devel/llvm20 , etc. >>> finish, but those builders not starting something new for a >>> notable time, can lead to huge RAM+SWAPSPACE usage for those >>> inactive builders for that time for USE_TMPFS=all without >>> TMPFS_BLACKLIST= in use. >> >> Further explorations in a ZFS context indicate that the following >> stick around in tmpfs for inactive builders for the USE_TMPFS=all >> type of context: >> >> /usr/local/poudriere/data/.m/*/*/usr/local/ (varying sizes, unlike the rest >> of the below) >> >> Note: the first * replaced text of the pattern [0-9][0-9] . >> In this case, 32 possibilities matched. >> >> The ones with lines having "32" as the count of matching lines >> in the later supporting material (largest size per to smallest): >> >> /usr/local/poudriere/data/.m/*/*/usr/bin/ >> /usr/local/poudriere/data/.m/*/*/var/cache/ >> /usr/local/poudriere/data/.m/*/*/usr/lib/ >> /usr/local/poudriere/data/.m/*/*/usr/lib32/ >> /usr/local/poudriere/data/.m/*/*/usr/share/ >> /usr/local/poudriere/data/.m/*/*/usr/include/ >> /usr/local/poudriere/data/.m/*/*/usr/sbin/ >> >> When a: >> >> /usr/local/poudriere/data/.m/*/*/usr/local/ >> >> has something like a lang/rust install that is large >> or even huge, the inactive builder is also individually >> large or huge as well. But there is no uniform "32" >> [or whatever] multiplier. My test did not happen to >> generate such an example usr/local/ . > > Well, rust's 1 GiBytes or so does appear to show up. Huge > compared to 1 GiByte or installing multiple large packages > into a usr/local/ did not happen but can happen depending > on what all is being built. > >> For reference material from my test: >> (The below avoids usr/src/ and usr/ports/ .) >> (/ref/ is not replaced, unlike /[0-9][0-9]/ becoming /*/ .) >> >> # du -xsAm >> /usr/local/poudriere/data/.m/*-default/*/[uvw]*/*[a-bd-rt-zA_Z0-9]/ | sed -e >> 's@/[0-9][0-9]/@/*/@' | sort -k1,2 | uniq -c | sort -n -k2,3 | tail -40 ; pou >> driere status -b >> 32 24 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/sbin/ >> 1 27 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/include/ >> 32 27 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/include/ >> 1 29 /usr/local/poudriere/data/.m/official-amd64-default/*/var/db/ >> 3 51 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 54 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 3 62 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 63 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 65 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 82 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/share/ >> 32 82 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/share/ >> 1 122 /usr/local/poudriere/data/.m/official-amd64-default/*/wrkdirs/usr/ >> 1 129 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib32/ >> 32 129 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib32/ >> 1 259 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib/ >> 32 259 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib/ >> 1 262 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 2 265 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 270 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 272 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 288 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 332 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 339 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 355 /usr/local/poudriere/data/.m/official-amd64-default/ref/var/cache/ >> 32 355 /usr/local/poudriere/data/.m/official-amd64-default/*/var/cache/ >> 1 379 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/bin/ >> 32 379 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/bin/ >> 1 381 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 423 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 446 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 462 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 520 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 522 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 532 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 3 546 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 548 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 657 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 794 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 1073 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> 1 1393 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ >> =>> [official-amd64-default] [2025-03-09_11h09m56s] [parallel_build] Time: >> 01:00:03 >> Queued: 450 Inspected: 0 Ignored: 2 Built: 440 Failed: 1 Skipped: 5 >> Fetched: 0 Remaining: 2 >> ID TOTAL ORIGIN PKGNAME PHASE TIME >> TMPFS CPU% MEM% >> [01] 00:00:07 x11-toolkits/wxgtk30@gtk3 | wx30-gtk3-3.0.5.1_6 configure >> 00:00:02 2.40 GiB 0.1% 0.1% >> =>> Logs: >> /usr/local/poudriere/data/logs/bulk/official-amd64-default/2025-03-09_11h09m56s I used an amd64 context for the below. # pkg rquery -e '%s >= 1073741824' -rFreeBSD '%sb (%sh): %o' | wc -l 63 # pkg rquery -e '%s >= 1073741824' -rFreeBSD '%sb (%sh): %o' | sort -nr 6312637268 (5.88GiB): x11-fonts/nerd-fonts 5028065806 (4.68GiB): cad/kicad-library-packages3d-devel 5014656473 (4.67GiB): cad/kicad-library-packages3d 4023994200 (3.75GiB): science/quantum-espresso-pseudopotentials 3759663181 (3.50GiB): print/texlive-docs 3523340789 (3.28GiB): games/0ad 3519431692 (3.28GiB): science/geant4 3498302532 (3.26GiB): java/intellij-ultimate 3456464186 (3.22GiB): games/ufoai-data 3419032211 (3.18GiB): print/texlive-texmf 3392802952 (3.16GiB): games/diaspora 3364341955 (3.13GiB): x11-fonts/iosevka 2849694498 (2.65GiB): devel/llvm-devel 2688064435 (2.50GiB): games/flightgear-data 2600325941 (2.42GiB): devel/pycharm-pro 2291266787 (2.13GiB): devel/jetbrains-clion 2265796966 (2.11GiB): x11-fonts/iosevka-comfy 2204358709 (2.05GiB): devel/jetbrains-goland 2125363178 (1.98GiB): devel/jetbrains-webstorm 2124118296 (1.98GiB): devel/jetbrains-phpstorm 2103384613 (1.96GiB): science/afni 2072316177 (1.93GiB): devel/jetbrains-rustrover 2031255963 (1.89GiB): graphics/piglit 2029504371 (1.89GiB): lang/ghc 1962446365 (1.83GiB): games/alienarena-data 1926448239 (1.79GiB): security/seclists 1910595369 (1.78GiB): games/urbanterror-data 1903938256 (1.77GiB): devel/llvm19 1856224925 (1.73GiB): devel/llvm18 1848424990 (1.72GiB): devel/llvm20 1697277047 (1.58GiB): devel/llvm17 1657044767 (1.54GiB): lang/rust-nightly 1654848172 (1.54GiB): lang/ghc810 1632416131 (1.52GiB): www/onlyoffice-documentserver 1615499118 (1.50GiB): devel/pycharm-ce 1606980295 (1.50GiB): editors/linux-r7-office 1596435146 (1.49GiB): devel/llvm14 1595739363 (1.49GiB): devel/rubymine 1552335509 (1.45GiB): science/openmolcas 1518593666 (1.41GiB): devel/llvm13 1496606231 (1.39GiB): devel/llvm16 1360483588 (1.27GiB): lang/linux-dotnet-sdk 1316472542 (1.23GiB): biology/cytoscape 1293585141 (1.20GiB): lang/ghc94 1247648556 (1.16GiB): lang/ghc92 1247211549 (1.16GiB): editors/lazarus-qt5-devel 1246863609 (1.16GiB): editors/lazarus-qt6-devel 1241195659 (1.16GiB): editors/lazarus-devel 1221647684 (1.14GiB): graphics/blender-doc 1214602305 (1.13GiB): japanese/dams 1196058036 (1.11GiB): games/pioneer 1195741547 (1.11GiB): x11-fonts/google-fonts 1194904485 (1.11GiB): lang/linux-dotnet11-sdk 1192270465 (1.11GiB): games/xonotic 1189280335 (1.11GiB): lang/dotnet 1172734857 (1.09GiB): games/quake-maps 1149121461 (1.07GiB): lang/rust 1133648487 (1.06GiB): editors/lazarus-qt5 1133294170 (1.06GiB): editors/lazarus-qt6 1126663720 (1.05GiB): editors/lazarus 1114054879 (1.04GiB): science/chrono 1085741063 (1.01GiB): games/redeclipse-data 1074153804 (1.00GiB): math/gap For reference, for >= 512 MiBytes, for >= 256 MiBytes, and for >= 128 MiBytes: # pkg rquery -e '%s >= 536870912' -rFreeBSD '%sb (%sh): %o' | wc -l 147 # pkg rquery -e '%s >= 268435456' -rFreeBSD '%sb (%sh): %o' | wc -l 331 # pkg rquery -e '%s >= 134217728' -rFreeBSD '%sb (%sh): %o' | wc -l 638 Of course, folks building there own that might add, say, line number debug information being kept, could have bigger figures for the same packages for the same platform. === Mark Millard marklmi at yahoo.com
