Re: FETCH_PACKAGES, mk.conf and make variable substitution time
On Wed, Dec 30, 2020 at 11:46:25PM +0100, Marc Espie wrote: > There aren't that many specific cases. But they exist and the consequences are subtle breakage in the ports infrastructure code. > Actually, you've got an example of bsd.port.mk dealing with things > for dwz, for instance. > > and the boostrap pseudo-flavor is another instance. I do not get what you're saying, can you elaborate? > This is definitely to be expected and is a design choice. What exactly? That FETCH_PACKAGES only works when passed on the command line? > the alternative would be to insist you install some binary packages before > building anything, and we don't do that. Not sure I follow, alternative to what? Yeah, we do not insist pkg_add'ing things before building other things, but we offer the possibility with FETCH_PACKAGES.
Re: FETCH_PACKAGES, mk.conf and make variable substitution time
On Wed, Dec 30, 2020 at 11:20:20PM +0100, Klemens Nanni wrote: > On Wed, Dec 30, 2020 at 09:30:57PM +, Stuart Henderson wrote: > > Oh wow, quite a deep rabbit hole to go down for this. I went for the > > simple approach: > > > > $ echo $PKG_PATH > > http://ftp.hostserver.de/pub/OpenBSD/snapshots/packages/%a/ > I don't use `-Dsnap' to ensure snapshot packages but rather to fetch > binary packages in the first place when ensuring dependencies during > ports building rather than building everything from source. > > I see how pkg_add(1) "special sequences" in PKG_PATH are much nicer than > the too simple installurl(5), but this is another topic. > > The mail's subject probably should not have mentioned FETCH_PACKAGES as > this issue is really about make(1)/bsd.port.mk(5) internals per se; > quirks/FETCH_PACKAGES is simply one specific case I encountered. > There aren't that many specific cases. Actually, you've got an example of bsd.port.mk dealing with things for dwz, for instance. and the boostrap pseudo-flavor is another instance. This is definitely to be expected and is a design choice. the alternative would be to insist you install some binary packages before building anything, and we don't do that.
Re: FETCH_PACKAGES, mk.conf and make variable substitution time
On Wed, Dec 30, 2020 at 09:30:57PM +, Stuart Henderson wrote: > Oh wow, quite a deep rabbit hole to go down for this. I went for the > simple approach: > > $ echo $PKG_PATH > http://ftp.hostserver.de/pub/OpenBSD/snapshots/packages/%a/ I don't use `-Dsnap' to ensure snapshot packages but rather to fetch binary packages in the first place when ensuring dependencies during ports building rather than building everything from source. I see how pkg_add(1) "special sequences" in PKG_PATH are much nicer than the too simple installurl(5), but this is another topic. The mail's subject probably should not have mentioned FETCH_PACKAGES as this issue is really about make(1)/bsd.port.mk(5) internals per se; quirks/FETCH_PACKAGES is simply one specific case I encountered.
Re: FETCH_PACKAGES, mk.conf and make variable substitution time
On 2020/12/30 21:51, Klemens Nanni wrote: > So the tech@ thread about make.1 improvements stems from an issue I have > with `make package' in devel/quirks and `FETCH_PACKAGES=-Dsnap' in > mk.conf(5): Oh wow, quite a deep rabbit hole to go down for this. I went for the simple approach: $ echo $PKG_PATH http://ftp.hostserver.de/pub/OpenBSD/snapshots/packages/%a/
FETCH_PACKAGES, mk.conf and make variable substitution time
So the tech@ thread about make.1 improvements stems from an issue I have
with `make package' in devel/quirks and `FETCH_PACKAGES=-Dsnap' in
mk.conf(5):
$ cat /etc/mk.conf
SUDO = doas
FETCH_PACKAGES =-Dsnap
PORTS_PRIVSEP = Yes
$ cd /usr/ports/devel/quirks
$ make package
===> Looking for quirks-3.504.tgz in $PKG_PATH - found
cp: /usr/ports/packages/amd64/cache/quirks-3.504.tgz: No such file or
directory
*** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2114
'/usr/ports/packages/amd64/no-arch/quirks-3.504.tgz': @cd /p/devel/quirks; g...)
*** Error 2 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2605
'_internal-package': @case X${_DEPENDS_CACHE} in X) _DEPENDS_CACHE=$(doas -...)
*** Error 2 in /p/devel/quirks
(/usr/ports/infrastructure/mk/bsd.port.mk:2584 'package': @lock=quirks-3.504;
export _LOCKS_HELD=" quirks-3)
Actual bsd.port.mk(5) line with failing cp(1) is 2129:
2114 # The real package
2115
2116 ${_pkg_cookie${_S}}:
2117 @${_PBUILD} install -d ${PACKAGE_REPOSITORY_MODE}
${_PKG_REPO${_S}} ${_TMP_REPO}
2118 . if ${FETCH_PACKAGES:L} != "no" &&
!defined(_TRIED_FETCHING_${_PACKAGE_COOKIE${_S}})
2119 @${_INSTALL_CACHE_REPO}
2120 @cd ${.CURDIR}; gotit=true; for p in ${_pkg${_S}}; do \
2121 if ! ${_PFETCH} ${MAKE} ${_CACHE_REPO}$$p; then \
2122 gotit=false; \
2123 break; \
2124 fi; \
2125 done; \
2126 if $$gotit; then \
2127 for p in ${_pkg${_S}}; do \
2128 s=${_CACHE_REPO}$$p; d=${_PKG_REPO${_S}}$$p; \
2129 ${_PBUILD} ln $$s $$d 2>/dev/null || ${_PBUILD}
cp -p $$s $$d; \
2130 done; \
2131 else \
2132 exec ${MAKE}
_TRIED_FETCHING_${_PACKAGE_COOKIE${_S}}=Yes _internal-package-only
_FETCH_RECURSE_HELPER=No; \
2133 fi
2134 . else
2135 @${_MAKE} ${_PACKAGE_COOKIE_DEPS}
That's because the code ends up with `gotit=true' even though there's
no package, be it fetched or built...
(`make -d c' and `make -d v' plus lots of grepping can help
understanding such scenarios.)
Anyway, quirks is already special, so what about explicitly setting
FETCH_PACKAGES in the port's Makefile:
Index: Makefile
===
RCS file: /cvs/ports/devel/quirks/Makefile,v
retrieving revision 1.1115
diff -u -p -r1.1115 Makefile
--- Makefile30 Dec 2020 08:44:38 - 1.1115
+++ Makefile30 Dec 2020 20:18:32 -
@@ -23,3 +23,6 @@ do-install:
${INSTALL_DATA} ${FILESDIR}/Quirks/ghc.pm
${PERLDIR}/OpenBSD/Quirks/ghc.pm
.include
+
+# build from source regardless of mk.conf(5)
+FETCH_PACKAGES = No
This still fails, but more ugly and later in the `${_CACHE_REPO}${_p}'
(bsd.port.mk line 2101) which is called above via line 2121:
$ make package
===> Looking for quirks-3.504.tgz in $PKG_PATH - Can't find No
found
cp: /usr/ports/packages/amd64/cache/quirks-3.504.tgz: No such file or
directory
*** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2114
'/usr/ports/packages/amd64/no-arch/quirks-3.504.tgz': @cd /p/devel/quirks; g...)
*** Error 2 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2605
'_internal-package': @case X${_DEPENDS_CACHE} in X) _DEPENDS_CACHE=$(doas -...)
*** Error 2 in /p/devel/quirks
(/usr/ports/infrastructure/mk/bsd.port.mk:2584 'package': @lock=quirks-3.504;
export _LOCKS_HELD=" quirks-3)
`FETCH_PACKAGES's value ends up as `pkg-name' argument to pkg_add(1)
("Can't find No") because bsd.port.mk(5) fails to handle this case.
This is because line
2118 . if ${FETCH_PACKAGES:L} != "no" &&
!defined(_TRIED_FETCHING_${_PACKAGE_COOKIE${_S}})
runs with the value of `FETCH_PACKAGES' at parse time, i.e. "-Dsnap"
from /etc/mk.conf, while line
2104 @if ${SETENV} [...] ${PKG_ADD} -I -x -n -q ${_PKG_ADD_FORCE} -r
-D installed -D downgrade ${FETCH_PACKAGES} ${@F}; then \
runs with its value at execution time, i.e. "No" from Makefile.
The only way to reliably build a quirks package is by overwriting
variables on the command line as those are set in stone before Makefiles
are pared and cannot be canged by them:
$ make FETCH_PACKAGES=No package
[...]
Create /usr/ports/packages/amd64/no-arch/quirks-3.504.tgz
Creating package quirks-3.504
Link to /usr/ports/packages/amd64/all/quirks-3.504.tgz
Link to /usr/ports/packages/amd64/ftp/quirks-3.504.tgz
Now this raises questions (for me):
- What is the expected hierarchy for Makefile, bsd.port.mk(5) and
mk.conf(5)? I'd expect Makefile to always have precedence and
actually work as such.
- Is this something worth fixing?
