Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
(Adding Colin Watson as he might have feedback on Ubuntu's uses of the various Release fields and on the likeliness to fix/change them) Hello, On Wed, 01 Feb 2023, Paul Gevers wrote: > > I fear I've probably forgotten most of these details, so please pardon > > me for this response… Wasn't the problem for Ubuntu that 'Pin: release > > foo' also applies to foo-proposed too? I think 'Pin: release > > foo-proposed' will work as intended though, right? Is the latter what > > we'll start generating with this? Seeing some example generated pins > > (before / after the patch) would be great to help reason about this. > > The patch also removes the "a=" from the pinning for the default release (at > 990) and I think that will break Ubuntu's setup as the packages from the > proposed pocket will suddenly satisfy this pin too. What you (Iain) > discussed above works for the pocket/foo-proposed part, but I think Raphael > needs the other part too. I fear that without additional options, we can't > really fix this. At this point I need neither part because I have added "Suite" names in Kali. But I think this behaviour is still entirely unlogical and will bite others in the future. I would suggest to: - apply the first hunk of my patch since this one seems to be fine - skip the change for the default release - document the limitation in --apt-default-release that it will only match packages from archives where "Suite" or "Archive" has this value (and maybe point back to this bug to find the context) > > I guess a test covering this for all of the Ubuntu, Debian & Kali cases > > would be helpful in terms of confidence both with this change and making > > any future changes here. The one thing I do remember is that it's hairy, > > like all the pinning stuff in autopkgtest. :-) > > Yes. In the same area; for Debian I once had a proposal to set the > Default-Release instead of using the pinning, but that broke Ubuntu's case. Indeed, as I shared in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1002819#10 Default-Release is actually implemented with the same code as the suggested pinning. > It would have reduced another hairy part of autopkgtest tremendously, where > a lot of sed/awk/grep-ing happens in the testbed to figure out which version > of the source to install for the test. But alas, autopkgtest needs to > support the Ubuntu archive. To be fair, the set of available fields to describe a repository are not very good, not very well documented and not easy to grasp. In my experience, Codename is the reference field that gives a canonical name. Suite is an alias that can move. And Archive is never used. The Ubuntu usage is not consistent with the above practice and they seem to use "Codename" as some sort of "Parent-Repository" or "Reference-Repository". There's no such field really and yet it would make sense to have one when you want to target all repositories that are compatible with a given release. I doubt that there's any chance that Ubuntu will fix their use of the Codename field but I'm putting Colin Watson in CC in case he can shed some light here. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Hi, Sorry it took so long. On 02-01-2022 01:43, Iain Lane wrote: On Sat, Jan 01, 2022 at 10:05:21PM +0100, Paul Gevers wrote: Hi Raphael, On 01-01-2022 21:37, Raphael Hertzog wrote: On Fri, 31 Dec 2021, Paul Gevers wrote: Otherwise I would like to suggest to create two entries, one with "Pin: release a=foo" and one with "Pin: release n=foo" so that we are sure to match on any of the 3 fields. I'll have to check and think about this. I remember that I had lots of issues with coming up with changes to autopkgtest that also worked for Ubuntu, as they use the same Codename for the real Suite and the *-proposed Suite (which they call pocket). I don't recall if that was with respect to pinning or other aspects of autopkgtest and it's requirement to manipulate where packages should be installed from. Before committing your proposal I need to understand that I'm not breaking existing valid configurations too. I saw a comment mentionning this, but it was related to the "--apt-pocket" option and I didn't change that part, which still uses the "a=foo" syntax. https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/lib/adt_testbed.py#L1263 Right, thanks for referencing that line as it has the bug number where the relevant information was. As the --pin-packages option will already have the *-pocket in the name, I think this would work for Ubuntu too. CC-ing Iain for a sanity check on our reasoning. Thanks for copying me on this. Julian might be a good one also. Done now. I fear I've probably forgotten most of these details, so please pardon me for this response… Wasn't the problem for Ubuntu that 'Pin: release foo' also applies to foo-proposed too? I think 'Pin: release foo-proposed' will work as intended though, right? Is the latter what we'll start generating with this? Seeing some example generated pins (before / after the patch) would be great to help reason about this. The patch also removes the "a=" from the pinning for the default release (at 990) and I think that will break Ubuntu's setup as the packages from the proposed pocket will suddenly satisfy this pin too. What you (Iain) discussed above works for the pocket/foo-proposed part, but I think Raphael needs the other part too. I fear that without additional options, we can't really fix this. I guess a test covering this for all of the Ubuntu, Debian & Kali cases would be helpful in terms of confidence both with this change and making any future changes here. The one thing I do remember is that it's hairy, like all the pinning stuff in autopkgtest. :-) Yes. In the same area; for Debian I once had a proposal to set the Default-Release instead of using the pinning, but that broke Ubuntu's case. It would have reduced another hairy part of autopkgtest tremendously, where a lot of sed/awk/grep-ing happens in the testbed to figure out which version of the source to install for the test. But alas, autopkgtest needs to support the Ubuntu archive. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Hey Paul, Raphael, happy new year to you, On Sat, Jan 01, 2022 at 10:05:21PM +0100, Paul Gevers wrote: > Hi Raphael, > > On 01-01-2022 21:37, Raphael Hertzog wrote: > > On Fri, 31 Dec 2021, Paul Gevers wrote: > > > > Otherwise I would like to suggest to create two entries, one with > > > > "Pin: release a=foo" and one with "Pin: release n=foo" so that > > > > we are sure to match on any of the 3 fields. > > > > > > I'll have to check and think about this. I remember that I had lots of > > > issues with coming up with changes to autopkgtest that also worked for > > > Ubuntu, as they use the same Codename for the real Suite and the > > > *-proposed > > > Suite (which they call pocket). I don't recall if that was with respect to > > > pinning or other aspects of autopkgtest and it's requirement to manipulate > > > where packages should be installed from. Before committing your proposal I > > > need to understand that I'm not breaking existing valid configurations > > > too. > > > > I saw a comment mentionning this, but it was related to the "--apt-pocket" > > option and I didn't change that part, which still uses the "a=foo" syntax. > > > > https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/lib/adt_testbed.py#L1263 > > Right, thanks for referencing that line as it has the bug number where the > relevant information was. As the --pin-packages option will already have the > *-pocket in the name, I think this would work for Ubuntu too. CC-ing Iain > for a sanity check on our reasoning. Thanks for copying me on this. Julian might be a good one also. I fear I've probably forgotten most of these details, so please pardon me for this response… Wasn't the problem for Ubuntu that 'Pin: release foo' also applies to foo-proposed too? I think 'Pin: release foo-proposed' will work as intended though, right? Is the latter what we'll start generating with this? Seeing some example generated pins (before / after the patch) would be great to help reason about this. I guess a test covering this for all of the Ubuntu, Debian & Kali cases would be helpful in terms of confidence both with this change and making any future changes here. The one thing I do remember is that it's hairy, like all the pinning stuff in autopkgtest. :-) Cheers, -- Iain Lane [ i...@orangesquash.org.uk ] Debian Developer [ la...@debian.org ] Ubuntu Developer [ la...@ubuntu.com ] signature.asc Description: PGP signature
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Hi Raphael, On 01-01-2022 21:37, Raphael Hertzog wrote: On Fri, 31 Dec 2021, Paul Gevers wrote: Otherwise I would like to suggest to create two entries, one with "Pin: release a=foo" and one with "Pin: release n=foo" so that we are sure to match on any of the 3 fields. I'll have to check and think about this. I remember that I had lots of issues with coming up with changes to autopkgtest that also worked for Ubuntu, as they use the same Codename for the real Suite and the *-proposed Suite (which they call pocket). I don't recall if that was with respect to pinning or other aspects of autopkgtest and it's requirement to manipulate where packages should be installed from. Before committing your proposal I need to understand that I'm not breaking existing valid configurations too. I saw a comment mentionning this, but it was related to the "--apt-pocket" option and I didn't change that part, which still uses the "a=foo" syntax. https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/lib/adt_testbed.py#L1263 Right, thanks for referencing that line as it has the bug number where the relevant information was. As the --pin-packages option will already have the *-pocket in the name, I think this would work for Ubuntu too. CC-ing Iain for a sanity check on our reasoning. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Hi, On Fri, 31 Dec 2021, Paul Gevers wrote: > > Otherwise I would like to suggest to create two entries, one with > > "Pin: release a=foo" and one with "Pin: release n=foo" so that > > we are sure to match on any of the 3 fields. > > I'll have to check and think about this. I remember that I had lots of > issues with coming up with changes to autopkgtest that also worked for > Ubuntu, as they use the same Codename for the real Suite and the *-proposed > Suite (which they call pocket). I don't recall if that was with respect to > pinning or other aspects of autopkgtest and it's requirement to manipulate > where packages should be installed from. Before committing your proposal I > need to understand that I'm not breaking existing valid configurations too. I saw a comment mentionning this, but it was related to the "--apt-pocket" option and I didn't change that part, which still uses the "a=foo" syntax. https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/lib/adt_testbed.py#L1263 And indeed in http://archive.ubuntu.com/ubuntu/dists/jammy-proposed/Release you have Suite: jammy-proposed Codename: jammy Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Hi Raphaël, Thanks for reporting. On 29-12-2021 11:44, Raphaël Hertzog wrote: Usage of --pin-packages=kali-dev=src:foo results in a file like this in /etc/apt/preferences.d/autopkgtest-kali-dev Package: foo Pin: release a=kali-dev Pin-Priority: 995 Unfortunately the "a=kali-dev" only matches on the "Suite" or "Archive" field in the Release file, and not on the "Codename" field (which in my caces was the only field available). I noticed that /etc/apt/preferences.d/autopkgtest-default-release uses another syntax that seems to cover more cases: Package: * Pin: release kali-rolling Pin-Priority: 990 However that syntax doesn't seem to be documented in apt_preferences. If it's correct and allows to check on either of the 3 fields, then we should likely use the same syntax in both files. Otherwise I would like to suggest to create two entries, one with "Pin: release a=foo" and one with "Pin: release n=foo" so that we are sure to match on any of the 3 fields. I'll have to check and think about this. I remember that I had lots of issues with coming up with changes to autopkgtest that also worked for Ubuntu, as they use the same Codename for the real Suite and the *-proposed Suite (which they call pocket). I don't recall if that was with respect to pinning or other aspects of autopkgtest and it's requirement to manipulate where packages should be installed from. Before committing your proposal I need to understand that I'm not breaking existing valid configurations too. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Control: tags -1 + patch
Control: user de...@kali.org
Control: usertag -1 + kali-patch
On Wed, 29 Dec 2021, Raphaël Hertzog wrote:
> I noticed that /etc/apt/preferences.d/autopkgtest-default-release uses
> another syntax that seems to cover more cases:
>
> Package: *
> Pin: release kali-rolling
> Pin-Priority: 990
>
> However that syntax doesn't seem to be documented in apt_preferences.
> If it's correct and allows to check on either of the 3 fields, then
> we should likely use the same syntax in both files.
I got confirmation from David Kalnischkies that this syntax is supported
and means exactly this:
12:54 buxy: yeah, its supported and means "Archive/Suite: or
Codename is X". One example in the manpage actually includes it (release
unstable), but it is never explained it seems. It is also the backbone of
commandline flag -t X. Code is in apt-pkg/versionmatch.cc, but its not
much to see.
So I recommend to use this same syntax in files generated for
--pin-packages.
Suggested patch is attached.
Cheers,
--
⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
>From 9946e61ed4821aa9c03d42393ea6887dc9337ea7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rapha=C3=ABl=20Hertzog?=
Date: Wed, 29 Dec 2021 13:23:27 +0100
Subject: [PATCH] Fix APT pinning for --pin-packages to also match on codenames
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
By using “Pin: release foo” instead of “Pin: release a=foo” we match
against all 3 relevant fields (Codename/Suite/Archive) whereas the
latter only matches against Suite and Archive.
Closes: #1002819
---
lib/adt_testbed.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/adt_testbed.py b/lib/adt_testbed.py
index b37eef4..225eb1f 100644
--- a/lib/adt_testbed.py
+++ b/lib/adt_testbed.py
@@ -1235,10 +1235,10 @@ Description: satisfy autopkgtest test dependencies
# prefer given packages from series, but make sure that other packages
# are taken from default release as much as possible
-script += 'printf "Package: $PKGS\\nPin: release a=%(release)s\\nPin-Priority: 995\\n" > /etc/apt/preferences.d/autopkgtest-%(release)s; ' % \
+script += 'printf "Package: $PKGS\\nPin: release %(release)s\\nPin-Priority: 995\\n" > /etc/apt/preferences.d/autopkgtest-%(release)s; ' % \
{'release': release}
for default in default_releases:
-script += 'printf "\nPackage: *\\nPin: release a=%(default)s\\nPin-Priority: 990\\n" >> /etc/apt/preferences.d/autopkgtest-%(release)s; ' % \
+script += 'printf "\nPackage: *\\nPin: release %(default)s\\nPin-Priority: 990\\n" >> /etc/apt/preferences.d/autopkgtest-%(release)s; ' % \
{'release': release, 'default': default}
self.check_exec(['sh', '-ec', script])
self._set_default_release()
--
2.34.1
Bug#1002819: autopkgtest: --pin-packages assumes that the release has a "Suite" name
Package: autopkgtest Version: 5.19 Severity: normal User: de...@kali.org Usertags: origin-kali X-Debbugs-Cc: hert...@debian.org Usage of --pin-packages=kali-dev=src:foo results in a file like this in /etc/apt/preferences.d/autopkgtest-kali-dev Package: foo Pin: release a=kali-dev Pin-Priority: 995 Unfortunately the "a=kali-dev" only matches on the "Suite" or "Archive" field in the Release file, and not on the "Codename" field (which in my caces was the only field available). I noticed that /etc/apt/preferences.d/autopkgtest-default-release uses another syntax that seems to cover more cases: Package: * Pin: release kali-rolling Pin-Priority: 990 However that syntax doesn't seem to be documented in apt_preferences. If it's correct and allows to check on either of the 3 fields, then we should likely use the same syntax in both files. Otherwise I would like to suggest to create two entries, one with "Pin: release a=foo" and one with "Pin: release n=foo" so that we are sure to match on any of the 3 fields. Cheers, -- System Information: Debian Release: bookworm/sid APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'oldoldstable'), (500, 'unstable'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 5.15.0-2-amd64 (SMP w/16 CPU threads) Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages autopkgtest depends on: ii apt-utils 2.3.13 ii libdpkg-perl1.21.1 ii procps 2:3.3.17-5 ii python3 3.9.8-1 ii python3-debian 0.1.42 Versions of packages autopkgtest recommends: ii autodep8 0.25 Versions of packages autopkgtest suggests: pn fakemachine pn lxc pn lxd ii ovmf 2021.11-1 pn ovmf-ia32 pn qemu-efi-aarch64 pn qemu-efi-arm pn qemu-system ii qemu-utils1:6.1+dfsg-8+b2 ii schroot 1.6.10-12 ii vmdb2 0.24-1 -- no debconf information
