On Mar 9, 2025, at 01:02, Mark Millard <[email protected]> wrote:
> In looking for why the builders used notable tmpfs space
> even when only one builder was left that was active,
> I discovered that, for example, each builder ends up
> with its own copy of /usr/local/poudriere/data/.m/*/*/usr/
> (and more) that does not end up being cleared out while
> the builder is inactive. This looked to be a systematic
> contribution to the tmpfs usage during times when various
> builders are inactive.
>
> df -m /usr/local/poudriere/data/.m/main-aarch64-default/*/*/ | sed -e
> 's@/[0-9][0-9]/@/*/@' | sort -k1,1 -k6,6 -k3,3 -k1,6 -u
> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue 1114846
> 498917 526741 49%
> /usr/local/poudriere/data/.m/main-aarch64-default/*/rescue
> /usr/local/poudriere/data/packages/main-aarch64-default/.building 1114846
> 498917 526741 49%
> /usr/local/poudriere/data/.m/main-aarch64-default/*/packages
> /usr/local/poudriere/data/packages/main-aarch64-default/.building 1114846
> 498917 526741 49%
> /usr/local/poudriere/data/.m/main-aarch64-default/ref/packages
> /usr/local/poudriere/jails/main-aarch64/rescue 1114846
> 498917 526741 49%
> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue
> /usr/ports/distfiles 1114846
> 498917 526741 49%
> /usr/local/poudriere/data/.m/main-aarch64-default/*/distfiles
> /usr/ports/distfiles 1114846
> 498917 526741 49%
> /usr/local/poudriere/data/.m/main-aarch64-default/ref/distfiles
> Filesystem 1M-blocks
> Used Avail Capacity Mounted on
> devfs 0
> 0 0 0% /usr/local/poudriere/data/.m/main-aarch64-default/*/dev
> devfs 0
> 0 0 0%
> /usr/local/poudriere/data/.m/main-aarch64-default/ref/dev
> procfs 0
> 0 0 0% /usr/local/poudriere/data/.m/main-aarch64-default/*/proc
> procfs 0
> 0 0 0%
> /usr/local/poudriere/data/.m/main-aarch64-default/ref/proc
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/01
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/02
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/03
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/04
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/05
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/06
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/07
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/08
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/09
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/10
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/11
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/12
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/13
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/14
> tmpfs 66539
> 1140 65398 2% /usr/local/poudriere/data/.m/main-aarch64-default/ref
>
> For the above, only ref/ and one other were active at the time.
> Imagine having 32 builders or 128 builders or even more with
> 1140 MiBytes for each inactive one. For the, above all the
> builders actually reported:
>
> Inspecting . . .: determining shlib requirements
>
> for each package build it tried. No actual builds were done.
>
> [I'll note that the world directory trees for the booted
> system and for the poudriere jail are from official
> PkgBase builds that were installed. Also, the system
> is UFS based, not ZFS based.]
>
> For reference (note the use of -x):
>
> # du -xsAm /usr/local/poudriere/data/.m/main-*-default/*/[uv][sa]r/ | sed -e
> 's@/[0-9][0-9]/@/*/@' | sort -k1,2 -u
> 376 /usr/local/poudriere/data/.m/main-aarch64-default/*/var/
> 376 /usr/local/poudriere/data/.m/main-aarch64-default/ref/var/
> 713 /usr/local/poudriere/data/.m/main-aarch64-default/*/usr/
> 713 /usr/local/poudriere/data/.m/main-aarch64-default/ref/usr/
>
> So 1089 MiBytes or so of the 1140 MiBytes for each such tmpfs
> is contributed overall by the combination of var/ and usr/ for
> the example.
>
> # poudriere jail -l
> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP
> PATH
> . . .
> main-aarch64 15.0-CURRENT aarch64 pkgbase 2025-02-12
> 22:49:27 /usr/local/poudriere/jails/main-aarch64
> . . .
>
> There is more to look into here, in that, historically, larger
> builders leave behind larger tmpfs usage until the next builder
> reuse (if any). Having a few of lang/rust , devel/llvm20 , etc.
> finish, but those builders not starting something new for a
> notable time, can lead to huge RAM+SWAPSPACE usage for those
> inactive builders for that time for USE_TMPFS=all without
> TMPFS_BLACKLIST= in use.
Further explorations in a ZFS context indicate that the following
stick around in tmpfs for inactive builders for the USE_TMPFS=all
type of context:
/usr/local/poudriere/data/.m/*/*/usr/local/ (varying sizes, unlike the rest of
the below)
Note: the first * replaced text of the pattern [0-9][0-9] .
In this case, 32 possibilities matched.
The ones with lines having "32" as the count of matching lines
in the later supporting material (largest size per to smallest):
/usr/local/poudriere/data/.m/*/*/usr/bin/
/usr/local/poudriere/data/.m/*/*/var/cache/
/usr/local/poudriere/data/.m/*/*/usr/lib/
/usr/local/poudriere/data/.m/*/*/usr/lib32/
/usr/local/poudriere/data/.m/*/*/usr/share/
/usr/local/poudriere/data/.m/*/*/usr/include/
/usr/local/poudriere/data/.m/*/*/usr/sbin/
When a:
/usr/local/poudriere/data/.m/*/*/usr/local/
has something like a lang/rust install that is large
or even huge, the inactive builder is also individually
large or huge as well. But there is no uniform "32"
[or whatever] multiplier. My test did not happen to
generate such an example usr/local/ .
For reference material from my test:
(The below avoids usr/src/ and usr/ports/ .)
(/ref/ is not replaced, unlike /[0-9][0-9]/ becoming /*/ .)
# du -xsAm /usr/local/poudriere/data/.m/*-default/*/[uvw]*/*[a-bd-rt-zA_Z0-9]/
| sed -e 's@/[0-9][0-9]/@/*/@' | sort -k1,2 | uniq -c | sort -n -k2,3 | tail
-40 ; pou
driere status -b
32 24 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/sbin/
1 27 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/include/
32 27 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/include/
1 29 /usr/local/poudriere/data/.m/official-amd64-default/*/var/db/
3 51 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 54 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
3 62 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 63 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 65 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 82 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/share/
32 82 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/share/
1 122 /usr/local/poudriere/data/.m/official-amd64-default/*/wrkdirs/usr/
1 129 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib32/
32 129 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib32/
1 259 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib/
32 259 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib/
1 262 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
2 265 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 270 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 272 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 288 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 332 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 339 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 355 /usr/local/poudriere/data/.m/official-amd64-default/ref/var/cache/
32 355 /usr/local/poudriere/data/.m/official-amd64-default/*/var/cache/
1 379 /usr/local/poudriere/data/.m/official-amd64-default/ref/usr/bin/
32 379 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/bin/
1 381 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 423 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 446 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 462 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 520 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 522 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 532 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
3 546 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 548 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 657 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 794 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 1073 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
1 1393 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
=>> [official-amd64-default] [2025-03-09_11h09m56s] [parallel_build] Time:
01:00:03
Queued: 450 Inspected: 0 Ignored: 2 Built: 440 Failed: 1 Skipped: 5
Fetched: 0 Remaining: 2
ID TOTAL ORIGIN PKGNAME PHASE TIME
TMPFS CPU% MEM%
[01] 00:00:07 x11-toolkits/wxgtk30@gtk3 | wx30-gtk3-3.0.5.1_6 configure
00:00:02 2.40 GiB 0.1% 0.1%
=>> Logs:
/usr/local/poudriere/data/logs/bulk/official-amd64-default/2025-03-09_11h09m56s
===
Mark Millard
marklmi at yahoo.com
