https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296241
Bug ID: 296241
Summary: pkgbase: "make update-packages" causes too much disk
churn
Product: Base System
Version: 16.0-CURRENT
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: bin
Assignee: [email protected]
Reporter: [email protected]
USER STORY
==========
As the maintainer of a private fork of FreeBSD, I would like to be able to
build pkgbase packages for each patch level of my fork, and be confident that
no packages have changed unnecessarily. One way to do that is to put REPODIR
on its own file system, snapshot that file system after every "make
update-packages" invocation, and compare snapshots' contents.
PROBLEM
=======
But if I do that, there's lots of disk churn between snapshots. "zfs diff"
shows that every package gets rewritten, even if its contents haven't changed.
That's unfortunate. It means that REPODIR's total disk consumption will
rapidly grow to many times the combined size of the packages.
ANALYSIS
========
The command that rewrites unchanged packages lies in share/mk/bsd.pkg.mk , as
shown below. That "cp" command copies the package into REPODIR even if it
already exists as is the same as in the source dir.
# Stage the packages from objdir into repodir.
stagepackages: .PHONY
mkdir -p ${_STAGEDIR}
cp ${_ALLPACKAGES:S/^/${PKG_NAME_PREFIX}-/:S/$/-${PKG_VERSION}.pkg/} \
${_STAGEDIR}
SOLUTIONS
=========
* One solution is ZFS dedup. It works right now, and a sysadmin can enable
dedup without any code changes. With a large recsize, it doesn't even take too
much RAM. Unfortunately, it does take some, and "zfs diff" will show a package
as being rewritten if it its contents are deduped.
* Another possible solution would be to replace the "cp" in stagepackages with
a command that only copies if the destination does not exist, or is different
from the source. That would probably work, though I haven't tried it yet.
* Modifying the makefiles to avoid even building packages whose inputs haven't
changed would be the canonical Make Way. But in this case, it looks pretty
hard.
CAVEATS
=======
The description above describes the behavior on 16.0-CURRENT. On 15.1-RELEASE
things are actually worse. There, "make update-packages" produces changed
packages every time. The problem seems to be that the packages' +MANIFEST
files contain mtime lines that depend on the build time. But that problem is
already fixed on 16.0-CURRENT.
--
You are receiving this mail because:
You are the assignee for the bug.
