On Mar 9, 2025, at 12:34, Mark Millard <[email protected]> wrote: > On Mar 9, 2025, at 01:02, Mark Millard <[email protected]> wrote: > >> In looking for why the builders used notable tmpfs space >> even when only one builder was left that was active, >> I discovered that, for example, each builder ends up >> with its own copy of /usr/local/poudriere/data/.m/*/*/usr/ >> (and more) that does not end up being cleared out while >> the builder is inactive. This looked to be a systematic >> contribution to the tmpfs usage during times when various >> builders are inactive. >> >> df -m /usr/local/poudriere/data/.m/main-aarch64-default/*/*/ | sed -e >> 's@/[0-9][0-9]/@/*/@' | sort -k1,1 -k6,6 -k3,3 -k1,6 -u >> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue 1114846 >> 498917 526741 49% >> /usr/local/poudriere/data/.m/main-aarch64-default/*/rescue >> /usr/local/poudriere/data/packages/main-aarch64-default/.building 1114846 >> 498917 526741 49% >> /usr/local/poudriere/data/.m/main-aarch64-default/*/packages >> /usr/local/poudriere/data/packages/main-aarch64-default/.building 1114846 >> 498917 526741 49% >> /usr/local/poudriere/data/.m/main-aarch64-default/ref/packages >> /usr/local/poudriere/jails/main-aarch64/rescue 1114846 >> 498917 526741 49% >> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue >> /usr/ports/distfiles 1114846 >> 498917 526741 49% >> /usr/local/poudriere/data/.m/main-aarch64-default/*/distfiles >> /usr/ports/distfiles 1114846 >> 498917 526741 49% >> /usr/local/poudriere/data/.m/main-aarch64-default/ref/distfiles >> Filesystem 1M-blocks >> Used Avail Capacity Mounted on >> devfs 0 >> 0 0 0% >> /usr/local/poudriere/data/.m/main-aarch64-default/*/dev >> devfs 0 >> 0 0 0% >> /usr/local/poudriere/data/.m/main-aarch64-default/ref/dev >> procfs 0 >> 0 0 0% >> /usr/local/poudriere/data/.m/main-aarch64-default/*/proc >> procfs 0 >> 0 0 0% >> /usr/local/poudriere/data/.m/main-aarch64-default/ref/proc >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/01 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/02 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/03 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/04 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/05 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/06 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/07 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/08 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/09 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/10 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/11 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/12 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/13 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/14 >> tmpfs 66539 >> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/ref >> >> For the above, only ref/ and one other were active at the time. >> Imagine having 32 builders or 128 builders or even more with >> 1140 MiBytes for each inactive one. For the, above all the >> builders actually reported: >> >> Inspecting . . .: determining shlib requirements >> >> for each package build it tried. No actual builds were done. >> >> [I'll note that the world directory trees for the booted >> system and for the poudriere jail are from official >> PkgBase builds that were installed. Also, the system >> is UFS based, not ZFS based.] >> >> For reference (note the use of -x): >> >> # du -xsAm /usr/local/poudriere/data/.m/main-*-default/*/[uv][sa]r/ | sed -e >> 's@/[0-9][0-9]/@/*/@' | sort -k1,2 -u >> 376 /usr/local/poudriere/data/.m/main-aarch64-default/*/var/ >> 376 /usr/local/poudriere/data/.m/main-aarch64-default/ref/var/ >> 713 /usr/local/poudriere/data/.m/main-aarch64-default/*/usr/ >> 713 /usr/local/poudriere/data/.m/main-aarch64-default/ref/usr/ >> >> So 1089 MiBytes or so of the 1140 MiBytes for each such tmpfs >> is contributed overall by the combination of var/ and usr/ for >> the example. >> >> # poudriere jail -l >> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP >> PATH >> . . . >> main-aarch64 15.0-CURRENT aarch64 pkgbase 2025-02-12 >> 22:49:27 /usr/local/poudriere/jails/main-aarch64 >> . . . >> >> There is more to look into here, in that, historically, larger >> builders leave behind larger tmpfs usage until the next builder >> reuse (if any). Having a few of lang/rust , devel/llvm20 , etc. >> finish, but those builders not starting something new for a >> notable time, can lead to huge RAM+SWAPSPACE usage for those >> inactive builders for that time for USE_TMPFS=all without >> TMPFS_BLACKLIST= in use. > > Further explorations in a ZFS context indicate that the following > stick around in tmpfs for inactive builders for the USE_TMPFS=all > type of context: > > /usr/local/poudriere/data/.m/*/*/usr/local/ (varying sizes, unlike the rest > of the below) > > Note: the first * replaced text of the pattern [0-9][0-9] . > In this case, 32 possibilities matched. > > The ones with lines having "32" as the count of matching lines > in the later supporting material (largest size per to smallest): > > /usr/local/poudriere/data/.m/*/*/usr/bin/ > /usr/local/poudriere/data/.m/*/*/var/cache/ > /usr/local/poudriere/data/.m/*/*/usr/lib/ > /usr/local/poudriere/data/.m/*/*/usr/lib32/ > /usr/local/poudriere/data/.m/*/*/usr/share/ > /usr/local/poudriere/data/.m/*/*/usr/include/ > /usr/local/poudriere/data/.m/*/*/usr/sbin/ > > When a: > > /usr/local/poudriere/data/.m/*/*/usr/local/ > > has something like a lang/rust install that is large > or even huge, the inactive builder is also individually > large or huge as well. But there is no uniform "32" > [or whatever] multiplier. My test did not happen to > generate such an example usr/local/ .
Well, rust's 1 GiBytes or so does appear to show up. Huge compared to 1 GiByte or installing multiple large packages into a usr/local/ did not happen but can happen depending on what all is being built. > For reference material from my test: > (The below avoids usr/src/ and usr/ports/ .) > (/ref/ is not replaced, unlike /[0-9][0-9]/ becoming /*/ .) > > # du -xsAm > /usr/local/poudriere/data/.m/*-default/*/[uvw]*/*[a-bd-rt-zA_Z0-9]/ | sed -e > 's@/[0-9][0-9]/@/*/@' | sort -k1,2 | uniq -c | sort -n -k2,3 | tail -40 ; pou > driere status -b > 32 24 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/sbin/ > 1 27 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/include/ > 32 27 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/include/ > 1 29 /usr/local/poudriere/data/.m/official-amd64-default/*/var/db/ > 3 51 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 54 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 3 62 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 63 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 65 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 82 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/share/ > 32 82 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/share/ > 1 122 /usr/local/poudriere/data/.m/official-amd64-default/*/wrkdirs/usr/ > 1 129 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib32/ > 32 129 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib32/ > 1 259 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib/ > 32 259 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib/ > 1 262 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 2 265 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 270 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 272 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 288 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 332 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 339 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 355 /usr/local/poudriere/data/.m/official-amd64-default/ref/var/cache/ > 32 355 /usr/local/poudriere/data/.m/official-amd64-default/*/var/cache/ > 1 379 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/bin/ > 32 379 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/bin/ > 1 381 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 423 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 446 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 462 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 520 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 522 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 532 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 3 546 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 548 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 657 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 794 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 1073 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > 1 1393 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/ > =>> [official-amd64-default] [2025-03-09_11h09m56s] [parallel_build] Time: > 01:00:03 > Queued: 450 Inspected: 0 Ignored: 2 Built: 440 Failed: 1 Skipped: 5 > Fetched: 0 Remaining: 2 > ID TOTAL ORIGIN PKGNAME PHASE TIME > TMPFS CPU% MEM% > [01] 00:00:07 x11-toolkits/wxgtk30@gtk3 | wx30-gtk3-3.0.5.1_6 configure > 00:00:02 2.40 GiB 0.1% 0.1% > =>> Logs: > /usr/local/poudriere/data/logs/bulk/official-amd64-default/2025-03-09_11h09m56s === Mark Millard marklmi at yahoo.com
