Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-25 Thread Johannes Schauer Marin Rodrigues
Hi,

Quoting Benjamin Drung (2022-01-25 21:01:11)
> > I find it interesting that somebody is using the --simulate option.  In
> > what scenario is it useful to have it?
> 
> We have a web UI to select packages (plus their version) and
> repositories that we want to add to the image that we built. Currently
> that list includes 37 packages, including the versioned kernel package,
> kernel modules, in-house software. That data is passed to our live
> image build system (which consists of bdebstrap YAML files plus many
> configs for /etc). In total the system uses variant minbase plus around
> 130 packages.
> 
> Before we switched to from our custom Shell script to use
> bdebstrap/mmdebstrap, we had a test script. This test script tried to
> "apt-get install -s" the packages to test the package selection. The
> test script had following benefits:
> 
> 1) It could detect incompatibilities between packages.
> 
> 2) It could detect if exactly one kernel package was installed. Since
> we built a live system, we need kernel (and we only want one to not
> waste space).
> 
> 3) It took only a few second to execute and could be used to quickly
> verify the package selection.
> 
> Now I am trying to get this feature working with bdebstrap/mmdebstrap.
> --simulate can do address point one and three. Point 2 can be done with a
> custom APT solver.

No need for a custom apt solver. If you just want to check whether apt installs
the stuff you expect, then you can make use of /usr/lib/apt/solvers/apt which
offers an EDSP interface to the apt solver. You could also use dose3 to ask
whether there exists a valid installation set but I think you want the exact
solution that apt would choose.  That's where /usr/lib/apt/solvers/apt comes
in. By feeding it your problem in the EDSP format [1] you can then parse the
result to find out what solution (if any) apt would go for. The advantage is,
that the input and output are in a well defined machine readable format and you
don't have to parse the output of apt-get --simulate which is intended for
human parsers.

Thanks!

cheers, josch


[1] /usr/share/doc/apt/external-dependency-solver-protocol.md.gz

signature.asc
Description: signature


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-25 Thread Benjamin Drung
Am Dienstag, dem 25.01.2022 um 17:58 +0100 schrieb Johannes Schauer
Marin Rodrigues:
> Quoting Benjamin Drung (2022-01-25 17:48:42)
> > > I do not yet understand why your "copy-in" essential hook where
> > > you copy
> > > bookworm.pref into /etc/apt/preferences.d inside the chroot
> > > doesn't solve
> > > this problem for you. Even though mmdebstrap doesn't run apt from
> > > inside
> > > the chroot anymore, it uses the configuration from /etc/apt
> > > inside the
> > > chroot.
> > 
> > "copy-in" works mostly for us, but it has two problems:
> > 
> > 1) "copy-in" is executed after the essential packages were
> > installed.
> > So bookworm.pref needs to be made available via aptopts for the
> > essential packages. In our in-house use case, we can live with this
> > drawback (since we rarely touch the essential packages).
> 
> Why not use "copy-in" as a setup-hook? Then the first "apt-get
> update" will
> pick up the configuration.

Thanks for this hint. We copy all of our configs for /etc in the
essential hook (for historic reasons). I will change that in-house.

> > 2) The essential hook is not executed with running with --simulate.
> > So the
> >    simulation would ignore the pinning file.
> 
> I find it interesting that somebody is using the --simulate option.
> In what
> scenario is it useful to have it?

We have a web UI to select packages (plus their version) and
repositories that we want to add to the image that we built. Currently
that list includes 37 packages, including the versioned kernel package,
kernel modules, in-house software. That data is passed to our live
image build system (which consists of bdebstrap YAML files plus many
configs for /etc). In total the system uses variant minbase plus around
130 packages.

Before we switched to from our custom Shell script to use
bdebstrap/mmdebstrap, we had a test script. This test script tried to
"apt-get install -s" the packages to test the package selection. The
test script had following benefits:

1) It could detect incompatibilities between packages.

2) It could detect if exactly one kernel package was installed. Since
we built a live system, we need kernel (and we only want one to not
waste space).

3) It took only a few second to execute and could be used to quickly
verify the package selection.

Now I am trying to get this feature working with bdebstrap/mmdebstrap.
--simulate can do address point one and three. Point 2 can be done with
a custom APT solver.

-- 
Benjamin Drung

Senior DevOps Engineer and Debian & Ubuntu Developer
Compute Platform Operations Cloud

IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland
E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498

Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning
Kettler, Arthur Mai, Britta Schmidt, Achim Weiß
Aufsichtsratsvorsitzender: Markus Kadelke


Member of United Internet


signature.asc
Description: This is a digitally signed message part


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-25 Thread Johannes Schauer Marin Rodrigues
Quoting Benjamin Drung (2022-01-25 17:48:42)
> > I do not yet understand why your "copy-in" essential hook where you copy
> > bookworm.pref into /etc/apt/preferences.d inside the chroot doesn't solve
> > this problem for you. Even though mmdebstrap doesn't run apt from inside
> > the chroot anymore, it uses the configuration from /etc/apt inside the
> > chroot.
> 
> "copy-in" works mostly for us, but it has two problems:
> 
> 1) "copy-in" is executed after the essential packages were installed.
> So bookworm.pref needs to be made available via aptopts for the
> essential packages. In our in-house use case, we can live with this
> drawback (since we rarely touch the essential packages).

Why not use "copy-in" as a setup-hook? Then the first "apt-get update" will
pick up the configuration.

> 2) The essential hook is not executed with running with --simulate. So the
>simulation would ignore the pinning file.

I find it interesting that somebody is using the --simulate option. In what
scenario is it useful to have it?

Thanks!

cheers, josch

signature.asc
Description: signature


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-25 Thread Benjamin Drung
Am Montag, dem 24.01.2022 um 17:23 +0100 schrieb Johannes Schauer Marin
Rodrigues:
> Quoting Benjamin Drung (2022-01-24 17:14:18)
> > > to make sure I understand your problem correctly: The mmdebstrap
> > > version in
> > > unstable is doing exactly what you want/expect but only the
> > > version in
> > > stable is not doing what you want?
> > 
> > Yes,
> 
> The difference you see between running the stable version with and
> without
> --simulate is that when you use --simulate, there is no apt inside
> the chroot
> and thus apt from the outside is used and that apt has access to
> different
> paths than the apt inside the chroot.
> 
> This is not the case anymore with mmdebstrap in unstable because
> there we never
> use apt inside the chroot [...]

That's a good change then. :)

> > but the mmdebstrap version in unstable requires one to specify the
> > full path
> > to the preference file.
> > 
> > It would be nice if I could set `Dir::Etc::preferencesparts` to
> > something like "$CURDIR/preferences.d/" without needed to parse
> > this
> > expression with Bash (since that would not work when specifying it
> > in a YAML
> > file for bdebstrap).
> 
> I do not yet understand why your "copy-in" essential hook where you
> copy
> bookworm.pref into /etc/apt/preferences.d inside the chroot doesn't
> solve this
> problem for you. Even though mmdebstrap doesn't run apt from inside
> the chroot
> anymore, it uses the configuration from /etc/apt inside the chroot.

"copy-in" works mostly for us, but it has two problems:

1) "copy-in" is executed after the essential packages were installed.
So bookworm.pref needs to be made available via aptopts for the
essential packages. In our in-house use case, we can live with this
drawback (since we rarely touch the essential packages).

2) The essential hook is not executed with running with --simulate. So
the simulation would ignore the pinning file.

-- 
Benjamin Drung

Senior DevOps Engineer and Debian & Ubuntu Developer
Compute Platform Operations Cloud

IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland
E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498

Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning
Kettler, Arthur Mai, Britta Schmidt, Achim Weiß
Aufsichtsratsvorsitzender: Markus Kadelke


Member of United Internet


signature.asc
Description: This is a digitally signed message part


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-24 Thread Johannes Schauer Marin Rodrigues
Quoting Benjamin Drung (2022-01-24 17:14:18)
> > to make sure I understand your problem correctly: The mmdebstrap version in
> > unstable is doing exactly what you want/expect but only the version in
> > stable is not doing what you want?
> 
> Yes,

The difference you see between running the stable version with and without
--simulate is that when you use --simulate, there is no apt inside the chroot
and thus apt from the outside is used and that apt has access to different
paths than the apt inside the chroot.

This is not the case anymore with mmdebstrap in unstable because there we never
use apt inside the chroot since apt included this fix:

https://salsa.debian.org/apt-team/apt/-/commit/544d81a26f8c97a2a45262aecbaef4a5b43fb325

That's also one of the reasons why mmdebstrap cannot be backported.

> but the mmdebstrap version in unstable requires one to specify the full path
> to the preference file.
> 
> It would be nice if I could set `Dir::Etc::preferencesparts` to
> something like "$CURDIR/preferences.d/" without needed to parse this
> expression with Bash (since that would not work when specifying it in a YAML
> file for bdebstrap).

I do not yet understand why your "copy-in" essential hook where you copy
bookworm.pref into /etc/apt/preferences.d inside the chroot doesn't solve this
problem for you. Even though mmdebstrap doesn't run apt from inside the chroot
anymore, it uses the configuration from /etc/apt inside the chroot.

Is that not working in your case somehow?

Thanks!

cheers, josch

signature.asc
Description: signature


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-24 Thread Benjamin Drung
Am Montag, dem 24.01.2022 um 16:54 +0100 schrieb Johannes Schauer Marin
Rodrigues:
> Quoting Benjamin Drung (2022-01-24 16:22:43)
> > I tested this in a minimal Debian schroot:
> > 
> > ```
> > apt -y install mmdebstrap
> > mkdir -p preferences.d
> > printf 'Package: *\nPin: release l=Debian,n=bookworm\nPin-Priority:
> > 400\n' > preferences.d/bookworm.pref
> > mmdebstrap -v --variant=minbase \
> >   --aptopt="Dir::Etc::preferencesparts \"$(pwd)/preferences.d/\"" \
> >   --include=python3-minimal/bullseye,python3 \
> >   --essential-hook="copy-in preferences.d/bookworm.pref
> > /etc/apt/preferences.d" \
> >   bullseye root.tar \
> >   "deb http://deb.debian.org/debian bullseye main" \
> >   "deb http://deb.debian.org/debian bookworm main"
> > ```
> > 
> > It fails on a Debian bullseye schroot, but succeeds on an unstable
> > schroot. Since the mmdebstrap version from unstable needs a newer
> > APT
> > version, there is no easy way for backporting.
> > 
> > Is setting `Dir::Etc::preferencesparts` to an absolute path the way
> > to
> > go forward? In this case it would be nice if I could set
> > `Dir::Etc::preferencesparts` to something like
> > "$CURDIR/preferences.d/"
> > and let mmdebstrap resolve it to the absolute path.
> 
> to make sure I understand your problem correctly: The mmdebstrap
> version in
> unstable is doing exactly what you want/expect but only the version
> in stable
> is not doing what you want?

Yes, but the mmdebstrap version in unstable requires one to specify the
full path to the preference file.

It would be nice if I could set `Dir::Etc::preferencesparts` to
something like "$CURDIR/preferences.d/" without needed to parse this
expression with Bash (since that would not work when specifying it in a
YAML file for bdebstrap).

-- 
Benjamin Drung

Senior DevOps Engineer and Debian & Ubuntu Developer
Compute Platform Operations Cloud

IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland
E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498

Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning
Kettler, Arthur Mai, Britta Schmidt, Achim Weiß
Aufsichtsratsvorsitzender: Markus Kadelke


Member of United Internet


signature.asc
Description: This is a digitally signed message part


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-24 Thread Johannes Schauer Marin Rodrigues
Quoting Benjamin Drung (2022-01-24 16:22:43)
> I tested this in a minimal Debian schroot:
> 
> ```
> apt -y install mmdebstrap
> mkdir -p preferences.d
> printf 'Package: *\nPin: release l=Debian,n=bookworm\nPin-Priority: 400\n' > 
> preferences.d/bookworm.pref
> mmdebstrap -v --variant=minbase \
>   --aptopt="Dir::Etc::preferencesparts \"$(pwd)/preferences.d/\"" \
>   --include=python3-minimal/bullseye,python3 \
>   --essential-hook="copy-in preferences.d/bookworm.pref 
> /etc/apt/preferences.d" \
>   bullseye root.tar \
>   "deb http://deb.debian.org/debian bullseye main" \
>   "deb http://deb.debian.org/debian bookworm main"
> ```
> 
> It fails on a Debian bullseye schroot, but succeeds on an unstable
> schroot. Since the mmdebstrap version from unstable needs a newer APT
> version, there is no easy way for backporting.
> 
> Is setting `Dir::Etc::preferencesparts` to an absolute path the way to
> go forward? In this case it would be nice if I could set
> `Dir::Etc::preferencesparts` to something like "$CURDIR/preferences.d/"
> and let mmdebstrap resolve it to the absolute path.

to make sure I understand your problem correctly: The mmdebstrap version in
unstable is doing exactly what you want/expect but only the version in stable
is not doing what you want?

Thanks!

cheers, josch

signature.asc
Description: signature


Bug#1004302: mmdebstrap: Unable to specify APT pinning (via Dir::Etc::preferencesparts)

2022-01-24 Thread Benjamin Drung
Package: mmdebstrap
Version: 0.7.5-2.2
Severity: normal

Hi,

I want to use mmdebstrap combined with APT pinning in a consistent way
(i.e. that mmdebstrap behaves the same in normal and dry-run mode), but
I failed to do so. To demonstrate the issue, I constructed a simple test
case: This test case defines bullseye and bookworm as mirrors and has an
APT pinning config to set the priority of bookworm packages below the
priority of bullseye packages. Without the pinning, the bookworm
packages will be newer and installed. With pinning, the bullseye
packages will be installed instead. Then install the package
`python3-minimal` from bullseye and let APT select the package version
for `python3`. Both packages need to be installed from the same release.
Otherwise APT will complain. That way the used pinning can be tested.

APT pinning is configured in `/etc/apt/preferences` and
`/etc/apt/preferences.d`. The location can be configured with the
settings `Dir::Etc::preferences` and `Dir::Etc::preferencesparts`. In
the test case, I am setting `Dir::Etc::preferencesparts`.

APT pinning `preferences.d/bookworm.pref`:

```
Package: *
Pin: release l=Debian,n=bookworm
Pin-Priority: 400
```

Here is the `pinning.yaml` configuration as YAML file for bdebstrap:

```
---
mmdebstrap:
  aptopts:
- Dir::Etc::preferencesparts "$CURDIR/preferences.d/"
  essential-hooks:
- copy-in preferences.d/bookworm.pref /etc/apt/preferences.d
  keyrings:
- /usr/share/keyrings/debian-archive-keyring.gpg
  mirrors:
- deb http://deb.debian.org/debian bullseye main
- deb http://deb.debian.org/debian bookworm main
  packages:
- python3-minimal/bullseye
- python3
  suite: bullseye
  target: root.tar
  variant: minbase
name: pinning-example
```

You have to set `$CURDIR` in the example above to the absolute path of your
current working directory.

1) If I specify a relative directory in `Dir::Etc::preferencesparts`, the
configuration is ignored.

2) If I set `Dir::Etc::preferencesparts` to an absolute path, it works in
`--dry-run`, but fails in the normal mode:

```
I: installing remaining packages inside the chroot...
Reading package lists...
Building dependency tree...
mawk is already the newest version (1.3.4.20200120-2).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python3 : PreDepends: python3-minimal (= 3.9.7-1) but 3.9.2-3 is to be 
installed
   Depends: python3.9 (>= 3.9.7-0~) but it is not going to be installed
   Depends: libpython3-stdlib (= 3.9.7-1) but it is not going to be 
installed
W: Unable to read /home/user/path/to/preferences.d/ - DirectoryExists (2: No 
such file or directory)
```

I tested this in a minimal Debian schroot:

```
apt -y install mmdebstrap
mkdir -p preferences.d
printf 'Package: *\nPin: release l=Debian,n=bookworm\nPin-Priority: 400\n' > 
preferences.d/bookworm.pref
mmdebstrap -v --variant=minbase \
  --aptopt="Dir::Etc::preferencesparts \"$(pwd)/preferences.d/\"" \
  --include=python3-minimal/bullseye,python3 \
  --essential-hook="copy-in preferences.d/bookworm.pref /etc/apt/preferences.d" 
\
  bullseye root.tar \
  "deb http://deb.debian.org/debian bullseye main" \
  "deb http://deb.debian.org/debian bookworm main"
```

It fails on a Debian bullseye schroot, but succeeds on an unstable
schroot. Since the mmdebstrap version from unstable needs a newer APT
version, there is no easy way for backporting.

Is setting `Dir::Etc::preferencesparts` to an absolute path the way to
go forward? In this case it would be nice if I could set
`Dir::Etc::preferencesparts` to something like "$CURDIR/preferences.d/"
and let mmdebstrap resolve it to the absolute path.

-- 
Benjamin Drung

Senior DevOps Engineer and Debian & Ubuntu Developer
Compute Platform Operations Cloud

IONOS SE | Revaler Str. 30 | 10245 Berlin | Deutschland
E-Mail: benjamin.dr...@ionos.com | Web: www.ionos.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498

Vorstand: Hüseyin Dogan, Dr. Martin Endreß, Claudia Frese, Henning
Kettler, Arthur Mai, Britta Schmidt, Achim Weiß
Aufsichtsratsvorsitzender: Markus Kadelke


Member of United Internet