Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Hi, Quoting Benjamin Drung (2022-01-25 21:01:11) > > I find it interesting that somebody is using the --simulate option. In > > what scenario is it useful to have it? > > We have a web UI to select packages (plus their version) and > repositories that we want to add to the image that we built. Currently > that list includes 37 packages, including the versioned kernel package, > kernel modules, in-house software. That data is passed to our live > image build system (which consists of bdebstrap YAML files plus many > configs for /etc). In total the system uses variant minbase plus around > 130 packages. > > Before we switched to from our custom Shell script to use > bdebstrap/mmdebstrap, we had a test script. This test script tried to > "apt-get install -s" the packages to test the package selection. The > test script had following benefits: > > 1) It could detect incompatibilities between packages. > > 2) It could detect if exactly one kernel package was installed. Since > we built a live system, we need kernel (and we only want one to not > waste space). > > 3) It took only a few second to execute and could be used to quickly > verify the package selection. > > Now I am trying to get this feature working with bdebstrap/mmdebstrap. > --simulate can do address point one and three. Point 2 can be done with a > custom APT solver. No need for a custom apt solver. If you just want to check whether apt installs the stuff you expect, then you can make use of /usr/lib/apt/solvers/apt which offers an EDSP interface to the apt solver. You could also use dose3 to ask whether there exists a valid installation set but I think you want the exact solution that apt would choose. That's where /usr/lib/apt/solvers/apt comes in. By feeding it your problem in the EDSP format [1] you can then parse the result to find out what solution (if any) apt would go for. The advantage is, that the input and output are in a well defined machine readable format and you don't have to parse the output of apt-get --simulate which is intended for human parsers. Thanks! cheers, josch [1] /usr/share/doc/apt/external-dependency-solver-protocol.md.gz signature.asc Description: signature
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Am Dienstag, dem 25.01.2022 um 17:58 +0100 schrieb Johannes Schauer Marin Rodrigues: > Quoting Benjamin Drung (2022-01-25 17:48:42) > > > I do not yet understand why your "copy-in" essential hook where > > > you copy > > > bookworm.pref into /etc/apt/preferences.d inside the chroot > > > doesn't solve > > > this problem for you. Even though mmdebstrap doesn't run apt from > > > inside > > > the chroot anymore, it uses the configuration from /etc/apt > > > inside the > > > chroot. > > > > "copy-in" works mostly for us, but it has two problems: > > > > 1) "copy-in" is executed after the essential packages were > > installed. > > So bookworm.pref needs to be made available via aptopts for the > > essential packages. In our in-house use case, we can live with this > > drawback (since we rarely touch the essential packages). > > Why not use "copy-in" as a setup-hook? Then the first "apt-get > update" will > pick up the configuration. Thanks for this hint. We copy all of our configs for /etc in the essential hook (for historic reasons). I will change that in-house. > > 2) The essential hook is not executed with running with --simulate. > > So the > > simulation would ignore the pinning file. > > I find it interesting that somebody is using the --simulate option. > In what > scenario is it useful to have it? We have a web UI to select packages (plus their version) and repositories that we want to add to the image that we built. Currently that list includes 37 packages, including the versioned kernel package, kernel modules, in-house software. That data is passed to our live image build system (which consists of bdebstrap YAML files plus many configs for /etc). In total the system uses variant minbase plus around 130 packages. Before we switched to from our custom Shell script to use bdebstrap/mmdebstrap, we had a test script. This test script tried to "apt-get install -s" the packages to test the package selection. The test script had following benefits: 1) It could detect incompatibilities between packages. 2) It could detect if exactly one kernel package was installed. Since we built a live system, we need kernel (and we only want one to not waste space). 3) It took only a few second to execute and could be used to quickly verify the package selection. Now I am trying to get this feature working with bdebstrap/mmdebstrap. --simulate can do address point one and three. Point 2 can be done with a custom APT solver. -- Benjamin Drung Senior DevOps Engineer and Debian & Ubuntu Developer Compute Platform Operations Cloud IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498 Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning Kettler, Arthur Mai, Britta Schmidt, Achim Weiß Aufsichtsratsvorsitzender: Markus Kadelke Member of United Internet signature.asc Description: This is a digitally signed message part
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Quoting Benjamin Drung (2022-01-25 17:48:42) > > I do not yet understand why your "copy-in" essential hook where you copy > > bookworm.pref into /etc/apt/preferences.d inside the chroot doesn't solve > > this problem for you. Even though mmdebstrap doesn't run apt from inside > > the chroot anymore, it uses the configuration from /etc/apt inside the > > chroot. > > "copy-in" works mostly for us, but it has two problems: > > 1) "copy-in" is executed after the essential packages were installed. > So bookworm.pref needs to be made available via aptopts for the > essential packages. In our in-house use case, we can live with this > drawback (since we rarely touch the essential packages). Why not use "copy-in" as a setup-hook? Then the first "apt-get update" will pick up the configuration. > 2) The essential hook is not executed with running with --simulate. So the >simulation would ignore the pinning file. I find it interesting that somebody is using the --simulate option. In what scenario is it useful to have it? Thanks! cheers, josch signature.asc Description: signature
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Am Montag, dem 24.01.2022 um 17:23 +0100 schrieb Johannes Schauer Marin Rodrigues: > Quoting Benjamin Drung (2022-01-24 17:14:18) > > > to make sure I understand your problem correctly: The mmdebstrap > > > version in > > > unstable is doing exactly what you want/expect but only the > > > version in > > > stable is not doing what you want? > > > > Yes, > > The difference you see between running the stable version with and > without > --simulate is that when you use --simulate, there is no apt inside > the chroot > and thus apt from the outside is used and that apt has access to > different > paths than the apt inside the chroot. > > This is not the case anymore with mmdebstrap in unstable because > there we never > use apt inside the chroot [...] That's a good change then. :) > > but the mmdebstrap version in unstable requires one to specify the > > full path > > to the preference file. > > > > It would be nice if I could set `Dir::Etc::preferencesparts` to > > something like "$CURDIR/preferences.d/" without needed to parse > > this > > expression with Bash (since that would not work when specifying it > > in a YAML > > file for bdebstrap). > > I do not yet understand why your "copy-in" essential hook where you > copy > bookworm.pref into /etc/apt/preferences.d inside the chroot doesn't > solve this > problem for you. Even though mmdebstrap doesn't run apt from inside > the chroot > anymore, it uses the configuration from /etc/apt inside the chroot. "copy-in" works mostly for us, but it has two problems: 1) "copy-in" is executed after the essential packages were installed. So bookworm.pref needs to be made available via aptopts for the essential packages. In our in-house use case, we can live with this drawback (since we rarely touch the essential packages). 2) The essential hook is not executed with running with --simulate. So the simulation would ignore the pinning file. -- Benjamin Drung Senior DevOps Engineer and Debian & Ubuntu Developer Compute Platform Operations Cloud IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498 Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning Kettler, Arthur Mai, Britta Schmidt, Achim Weiß Aufsichtsratsvorsitzender: Markus Kadelke Member of United Internet signature.asc Description: This is a digitally signed message part
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Quoting Benjamin Drung (2022-01-24 17:14:18) > > to make sure I understand your problem correctly: The mmdebstrap version in > > unstable is doing exactly what you want/expect but only the version in > > stable is not doing what you want? > > Yes, The difference you see between running the stable version with and without --simulate is that when you use --simulate, there is no apt inside the chroot and thus apt from the outside is used and that apt has access to different paths than the apt inside the chroot. This is not the case anymore with mmdebstrap in unstable because there we never use apt inside the chroot since apt included this fix: https://salsa.debian.org/apt-team/apt/-/commit/544d81a26f8c97a2a45262aecbaef4a5b43fb325 That's also one of the reasons why mmdebstrap cannot be backported. > but the mmdebstrap version in unstable requires one to specify the full path > to the preference file. > > It would be nice if I could set `Dir::Etc::preferencesparts` to > something like "$CURDIR/preferences.d/" without needed to parse this > expression with Bash (since that would not work when specifying it in a YAML > file for bdebstrap). I do not yet understand why your "copy-in" essential hook where you copy bookworm.pref into /etc/apt/preferences.d inside the chroot doesn't solve this problem for you. Even though mmdebstrap doesn't run apt from inside the chroot anymore, it uses the configuration from /etc/apt inside the chroot. Is that not working in your case somehow? Thanks! cheers, josch signature.asc Description: signature
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Am Montag, dem 24.01.2022 um 16:54 +0100 schrieb Johannes Schauer Marin Rodrigues: > Quoting Benjamin Drung (2022-01-24 16:22:43) > > I tested this in a minimal Debian schroot: > > > > ``` > > apt -y install mmdebstrap > > mkdir -p preferences.d > > printf 'Package: *\nPin: release l=Debian,n=bookworm\nPin-Priority: > > 400\n' > preferences.d/bookworm.pref > > mmdebstrap -v --variant=minbase \ > > --aptopt="Dir::Etc::preferencesparts \"$(pwd)/preferences.d/\"" \ > > --include=python3-minimal/bullseye,python3 \ > > --essential-hook="copy-in preferences.d/bookworm.pref > > /etc/apt/preferences.d" \ > > bullseye root.tar \ > > "deb http://deb.debian.org/debian bullseye main" \ > > "deb http://deb.debian.org/debian bookworm main" > > ``` > > > > It fails on a Debian bullseye schroot, but succeeds on an unstable > > schroot. Since the mmdebstrap version from unstable needs a newer > > APT > > version, there is no easy way for backporting. > > > > Is setting `Dir::Etc::preferencesparts` to an absolute path the way > > to > > go forward? In this case it would be nice if I could set > > `Dir::Etc::preferencesparts` to something like > > "$CURDIR/preferences.d/" > > and let mmdebstrap resolve it to the absolute path. > > to make sure I understand your problem correctly: The mmdebstrap > version in > unstable is doing exactly what you want/expect but only the version > in stable > is not doing what you want? Yes, but the mmdebstrap version in unstable requires one to specify the full path to the preference file. It would be nice if I could set `Dir::Etc::preferencesparts` to something like "$CURDIR/preferences.d/" without needed to parse this expression with Bash (since that would not work when specifying it in a YAML file for bdebstrap). -- Benjamin Drung Senior DevOps Engineer and Debian & Ubuntu Developer Compute Platform Operations Cloud IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498 Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning Kettler, Arthur Mai, Britta Schmidt, Achim Weiß Aufsichtsratsvorsitzender: Markus Kadelke Member of United Internet signature.asc Description: This is a digitally signed message part
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Quoting Benjamin Drung (2022-01-24 16:22:43) > I tested this in a minimal Debian schroot: > > ``` > apt -y install mmdebstrap > mkdir -p preferences.d > printf 'Package: *\nPin: release l=Debian,n=bookworm\nPin-Priority: 400\n' > > preferences.d/bookworm.pref > mmdebstrap -v --variant=minbase \ > --aptopt="Dir::Etc::preferencesparts \"$(pwd)/preferences.d/\"" \ > --include=python3-minimal/bullseye,python3 \ > --essential-hook="copy-in preferences.d/bookworm.pref > /etc/apt/preferences.d" \ > bullseye root.tar \ > "deb http://deb.debian.org/debian bullseye main" \ > "deb http://deb.debian.org/debian bookworm main" > ``` > > It fails on a Debian bullseye schroot, but succeeds on an unstable > schroot. Since the mmdebstrap version from unstable needs a newer APT > version, there is no easy way for backporting. > > Is setting `Dir::Etc::preferencesparts` to an absolute path the way to > go forward? In this case it would be nice if I could set > `Dir::Etc::preferencesparts` to something like "$CURDIR/preferences.d/" > and let mmdebstrap resolve it to the absolute path. to make sure I understand your problem correctly: The mmdebstrap version in unstable is doing exactly what you want/expect but only the version in stable is not doing what you want? Thanks! cheers, josch signature.asc Description: signature
Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)
Package: mmdebstrap Version: 0.7.5-2.2 Severity: normal Hi, I want to use mmdebstrap combined with APT pinning in a consistent way (i.e. that mmdebstrap behaves the same in normal and dry-run mode), but I failed to do so. To demonstrate the issue, I constructed a simple test case: This test case defines bullseye and bookworm as mirrors and has an APT pinning config to set the priority of bookworm packages below the priority of bullseye packages. Without the pinning, the bookworm packages will be newer and installed. With pinning, the bullseye packages will be installed instead. Then install the package `python3-minimal` from bullseye and let APT select the package version for `python3`. Both packages need to be installed from the same release. Otherwise APT will complain. That way the used pinning can be tested. APT pinning is configured in `/etc/apt/preferences` and `/etc/apt/preferences.d`. The location can be configured with the settings `Dir::Etc::preferences` and `Dir::Etc::preferencesparts`. In the test case, I am setting `Dir::Etc::preferencesparts`. APT pinning `preferences.d/bookworm.pref`: ``` Package: * Pin: release l=Debian,n=bookworm Pin-Priority: 400 ``` Here is the `pinning.yaml` configuration as YAML file for bdebstrap: ``` --- mmdebstrap: aptopts: - Dir::Etc::preferencesparts "$CURDIR/preferences.d/" essential-hooks: - copy-in preferences.d/bookworm.pref /etc/apt/preferences.d keyrings: - /usr/share/keyrings/debian-archive-keyring.gpg mirrors: - deb http://deb.debian.org/debian bullseye main - deb http://deb.debian.org/debian bookworm main packages: - python3-minimal/bullseye - python3 suite: bullseye target: root.tar variant: minbase name: pinning-example ``` You have to set `$CURDIR` in the example above to the absolute path of your current working directory. 1) If I specify a relative directory in `Dir::Etc::preferencesparts`, the configuration is ignored. 2) If I set `Dir::Etc::preferencesparts` to an absolute path, it works in `--dry-run`, but fails in the normal mode: ``` I: installing remaining packages inside the chroot... Reading package lists... Building dependency tree... mawk is already the newest version (1.3.4.20200120-2). Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: python3 : PreDepends: python3-minimal (= 3.9.7-1) but 3.9.2-3 is to be installed Depends: python3.9 (>= 3.9.7-0~) but it is not going to be installed Depends: libpython3-stdlib (= 3.9.7-1) but it is not going to be installed W: Unable to read /home/user/path/to/preferences.d/ - DirectoryExists (2: No such file or directory) ``` I tested this in a minimal Debian schroot: ``` apt -y install mmdebstrap mkdir -p preferences.d printf 'Package: *\nPin: release l=Debian,n=bookworm\nPin-Priority: 400\n' > preferences.d/bookworm.pref mmdebstrap -v --variant=minbase \ --aptopt="Dir::Etc::preferencesparts \"$(pwd)/preferences.d/\"" \ --include=python3-minimal/bullseye,python3 \ --essential-hook="copy-in preferences.d/bookworm.pref /etc/apt/preferences.d" \ bullseye root.tar \ "deb http://deb.debian.org/debian bullseye main" \ "deb http://deb.debian.org/debian bookworm main" ``` It fails on a Debian bullseye schroot, but succeeds on an unstable schroot. Since the mmdebstrap version from unstable needs a newer APT version, there is no easy way for backporting. Is setting `Dir::Etc::preferencesparts` to an absolute path the way to go forward? In this case it would be nice if I could set `Dir::Etc::preferencesparts` to something like "$CURDIR/preferences.d/" and let mmdebstrap resolve it to the absolute path. -- Benjamin Drung Senior DevOps Engineer and Debian & Ubuntu Developer Compute Platform Operations Cloud IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498 Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning Kettler, Arthur Mai, Britta Schmidt, Achim Weiß Aufsichtsratsvorsitzender: Markus Kadelke Member of United Internet
