Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Philip Hands]

Control: reassign -1 apt-setup-udeb
Control: fixed -1 1:0.169

Hi,

I just had a look at this, and it seems to me that this was fixed in
apt-setup-udeb 0.169, but the version in the released (Debian 11)
installer is only at 0.166, so does not include the fix.

Looking at the syslog in this bug, one can see:

  apt-setup-udeb 1:0.166

which is the version in the release, and is from 2021-07-23.

The thing that fixes the bug is:

  https://salsa.debian.org/installer-team/apt-setup/-/merge_requests/4

which was merged on 2022-01-29, then released as part of 1:0.169.

I've reproduced the failure with the release version of D-I, and failed
to reproduce it with yesterday's daily image (where one sees the
installation of the c-certificates package go past just after selecting
the mirror), so it really looks to have been fixed already.

If you want to try that for yourself, the daily images can be found
here:

  
https://cdimage.debian.org/cdimage/daily-builds/sid_d-i/arch-latest/amd64/iso-cd/debian-testing-amd64-netinst.iso

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Cyril Brulebois]

Richard Hector  (2022-07-24):
> Oh - uncompressed, it made it into the BTS, but not to the list. Here's a
> compressed version.

Thanks.

debootstrap uses the ISO's contents, so https isn't noticed at this point
(final argument):

Jul 23 01:03:18 debootstrap: /usr/sbin/debootstrap --components=main 
--debian-installer --resolve-deps --no-check-gpg bullseye /target file:///cdrom/

Later:

Jul 23 01:07:13 apt-setup: Identifying...
Jul 23 01:07:13 apt-setup: [5f70f43faa4e30b11b269f8c73178e29-2]
Jul 23 01:07:13 apt-setup: Scanning disc for index files...
Jul 23 01:07:13 apt-setup: Found 1 package indexes, 0 source indexes, 1 
translation indexes and 0 signatures
Jul 23 01:07:13 apt-setup: This disc is called:
Jul 23 01:07:13 apt-setup: 'Debian GNU/Linux 11.4.0 _Bullseye_ - Official 
amd64 NETINST 20220709-10:31'
Jul 23 01:07:13 apt-setup: Copying package lists...
Jul 23 01:07:13 apt-setup: ^MReading Package Indexes... 0%^M
Jul 23 01:07:13 apt-setup: ^MReading Package Indexes... 0%^M
Jul 23 01:07:13 apt-setup: ^MReading Package Indexes... Done^M
Jul 23 01:07:13 apt-setup: ^MReading Translation Indexes... 0%^M
Jul 23 01:07:13 apt-setup: ^MReading Translation Indexes... Done^M
Jul 23 01:07:13 apt-setup: Writing new source list
Jul 23 01:07:13 apt-setup: Source list entries for this disc are:
Jul 23 01:07:13 apt-setup: deb cdrom:[Debian GNU/Linux 11.4.0 _Bullseye_ - 
Official amd64 NETINST 20220709-10:31]/ bullseye main
Jul 23 01:07:13 apt-setup: Repeat this process for the rest of the CDs in 
your set.
Jul 23 01:07:45 choose-mirror[24148]: DEBUG: command: wget --no-verbose 
https://deb.debian.org/debian/dists/bullseye/Release -O - | grep -E 
'^(Suite|Codename|Architectures):'
Jul 23 01:07:45 choose-mirror[24148]: DEBUG: command: wget --no-verbose 
https://deb.debian.org/debian/dists/stable/Release -O - | grep -E 
'^(Suite|Codename|Architectures):'
Jul 23 01:07:46 choose-mirror[24148]: INFO: suite/codename set to: 
stable/bullseye
Jul 23 01:07:46 choose-mirror[24148]: DEBUG: command: wget --no-verbose 
https://deb.debian.org/debian//dists/bullseye/main/binary-amd64/Release -O - | 
grep ^Architecture:
Jul 23 01:08:12 apt-setup: dpkg-divert: warning: diverting file 
'/sbin/start-stop-daemon' from an Essential package with rename is dangerous, 
use --no-rename
Jul 23 01:08:13 in-target: Err:1 https://deb.debian.org/debian bullseye 
InRelease
Jul 23 01:08:13 in-target:   Certificate verification failed: The 
certificate is NOT trusted. The certificate issuer is unknown.  Could not 
handshake: Error in the certificate verification. [IP: 2a04:4e42:27::644 443]

I think the choose-mirror calls come from apt-setup's generators/50mirror
(after generators/40cdrom and generators/41cdset), and that one is supposed
to know about ca-certificates:
  
https://salsa.debian.org/installer-team/apt-setup/-/blob/master/generators/50mirror#L233-245

I suppose the in-target calls might be from apt-setup-verify, called later:
  
https://salsa.debian.org/installer-team/apt-setup/-/blob/master/generators/50mirror#L264

If you want to help troubleshoot that further, checking the debconf
exchanges could be interesting. I think we support setting
DEBCONF_DEBUG=developer on the kernel command line, which should make
debconf queries/answers (as triggered by db_get and friends) appear in the
syslog. Past $self seems to agree:
  https://mraw.org/blog/2012/12/23/d-i_hacking_recipe_3/


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Geert Stappers]

On Sun, Jul 24, 2022 at 12:15:24AM +1200, Richard Hector wrote:
> On 23/07/22 23:01, Cyril Brulebois wrote:
> 
> > As mentioned by Julien, getting the installer's syslog (compressed, to
> > make sure it reaches the mailing list) would help understand what's
> > going on.
> 
> Oh - uncompressed, it made it into the BTS,
 
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1015887;filename=syslog;msg=27

Jul 23 01:08:13 in-target: Err:1 https://deb.debian.org/debian bullseye 
InRelease
Jul 23 01:08:13 in-target:   Certificate verification failed: The certificate 
is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error 
in the certificate verification. [IP: 2a04:4e42:27::644 443]
Jul 23 01:08:13 in-target: Reading package lists...
Jul 23 01:08:13 in-target: 
Jul 23 01:08:13 in-target: W: 
https://deb.debian.org/debian/dists/bullseye/InRelease: No system certificates 
available. Try installing ca-certificates.
Jul 23 01:08:13 in-target: W: Failed to fetch 
https://deb.debian.org/debian/dists/bullseye/InRelease  Certificate 
verification failed: The certificate is NOT trusted. The certificate issuer is 
unknown.  Could not handshake: Error in the certificate verification. [IP: 
2a04:4e42:27::644 443]
Jul 23 01:08:13 in-target: W: Some index files failed to download. They have 
been ignored, or old ones used instead.
Jul 23 01:08:13 apt-setup: dpkg-divert: warning: diverting file 
'/sbin/start-stop-daemon' from an Essential package with rename is dangerous, 
use --no-rename
Jul 23 01:08:14 in-target: Err:1 https://deb.debian.org/debian bullseye 
InRelease
Jul 23 01:08:14 in-target:   Certificate verification failed: The certificate 
is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error 
in the certificate verification. [IP: 2a04:4e42:27::644 443]
Jul 23 01:08:14 in-target: Reading package lists...
Jul 23 01:08:14 in-target: 
Jul 23 01:08:14 in-target: W: 
https://deb.debian.org/debian/dists/bullseye/InRelease: No system certificates 
available. Try installing ca-certificates.
Jul 23 01:08:14 in-target: W: Failed to fetch 
https://deb.debian.org/debian/dists/bullseye/InRelease  Certificate 
verification failed: The certificate is NOT trusted. The certificate issuer is 
unknown.  Could not handshake: Error in the certificate verification. [IP: 
2a04:4e42:27::644 443]
Jul 23 01:08:14 in-target: W: Some index files failed to download. They have 
been ignored, or old ones used instead.


no traces of manual install of ca-certificates found by me.


Regards
Geert Stappers
Failed to explain that httpS is NOT needed for apt.
Agrees that it is nice to have ca-certificates installed.
-- 
Silence is hard to parse

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Richard Hector]

On 23/07/22 23:01, Cyril Brulebois wrote:


As mentioned by Julien, getting the installer's syslog (compressed, to
make sure it reaches the mailing list) would help understand what's
going on.


Oh - uncompressed, it made it into the BTS, but not to the list. Here's 
a compressed version.


Cheers,
Richard

syslog.gz
Description: application/gzip

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Cyril Brulebois]

Control: severity -1 important

Richard Hector  (2022-07-23):
> On 23/07/22 18:07, Geert Stappers wrote:
> > Control: severity -1 wishlist
> 
> Why? Because there's a workaround? Is everyone expected to be able to find
> that workaround?
> 
> https is an option provided in the installer, that apparently doesn't work
> (at least with the netinst installer), and it's not immediately clear why.

That's definitely something that ought to work, fixing severity.

(I do test installation using HTTPS for all releases, even if that's
using the netboot-gtk mini.iso, seeding repository parameters via the
kernel command line; so HTTPS support should not be *horribly* broken.)

We even have code to install apt-transport-https conditionally (since
that feature was merged into apt proper a while back), see:
  
https://salsa.debian.org/installer-team/debootstrap/-/blob/master/scripts/debian-common#L30-42

I remember having to patch a few components to make sure it would work
for all installation images, when support was implemented in the first
place.

As mentioned by Julien, getting the installer's syslog (compressed, to
make sure it reaches the mailing list) would help understand what's
going on.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Julien Cristau]

On Sat, Jul 23, 2022 at 03:49:55PM +1200, Richard Hector wrote:
> Package: debian-installer
> Severity: important
> 
> Dear Maintainer,
> 
> Using netinst bullseye 11.4 installer:
> 
> https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.4.0-amd64-netinst.iso
> 
> I chose to add a network mirror, using https, and the default
> 'deb.debian.org'.
> 
> I used (non-graphical) Expert Mode.
> 
> The problem first showed up when tasksel only displayed 'standard system
> utilities'. When I went ahead with that, the next screen was a red
> 'Installation step failed' screen.
> 
> The log on tty4 showed various dependency problems.
> 
> I tried to 'chroot /target' and 'apt update', which showed certificate
> problems. I then ran 'apt install ca-certificates', which worked
> (installing from the cd image?), after which 'apt update' worked, and I
> was also able to continue successfully with the installer.
> 
> I was able to reproduce this in a (kvm/qemu) VM (which is where I
> confirmed my steps); the original problem was on an HP Thin Client
> (t520). In both cases only 8G of storage was available.
> 
> It all works fine using http for the mirror.
> 
> I'm happy to do further testing with the VM; the thin client is less
> convenient as it has a job to do.
> 
Please attach syslog from the installer.

Cheers,
Julien

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Richard Hector]

On 23/07/22 18:07, Geert Stappers wrote:

Control: severity -1 wishlist


Why? Because there's a workaround? Is everyone expected to be able to 
find that workaround?


https is an option provided in the installer, that apparently doesn't 
work (at least with the netinst installer), and it's not immediately 
clear why.


Essentially, I think it's a showstopper for anyone who doesn't know how 
to investigate further.



It all works fine using http for the mirror.


And the archive mirror content is secured by checksums and signatures.


The point being that https isn't necessary? A different issue, I think.


I'm happy to do further testing with the VM; the thin client is less
convenient as it has a job to do.


Another job that will help: Find other bug reports that ask for installing
ca-certificates.  Yeah, I recall have I seen such requests before.


Not sure how to do that. The BTS UI doesn't seem to allow searching on 
the content of bug discussions; only subject and other metadata. I can't 
see any other debian-installer bugs that mention ca-certificates in the 
subject.


Cheers,
Richard

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Geert Stappers]

Control: severity -1 wishlist

On Sat, Jul 23, 2022 at 03:49:55PM +1200, Richard Hector wrote:
> Dear Maintainer,
> 
> Using netinst bullseye 11.4 installer:
> 
> https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.4.0-amd64-netinst.iso
> 
> I chose to add a network mirror, using https, and the default
> 'deb.debian.org'.
> 
> I used (non-graphical) Expert Mode.
> 
> The problem first showed up when tasksel only displayed 'standard system
> utilities'. When I went ahead with that, the next screen was a red
> 'Installation step failed' screen.
> 
> The log on tty4 showed various dependency problems.
> 
> I tried to 'chroot /target' and 'apt update', which showed certificate
> problems. I then ran 'apt install ca-certificates', which worked
> (installing from the cd image?), after which 'apt update' worked, and I
> was also able to continue successfully with the installer.
> 
> I was able to reproduce this in a (kvm/qemu) VM (which is where I
> confirmed my steps); the original problem was on an HP Thin Client
> (t520). In both cases only 8G of storage was available.
> 
> It all works fine using http for the mirror.

And the archive mirror content is secured by checksums and signatures.

 
> I'm happy to do further testing with the VM; the thin client is less
> convenient as it has a job to do.

Another job that will help: Find other bug reports that ask for installing
ca-certificates.  Yeah, I recall have I seen such requests before.


Groeten
Geert Stappers
-- 
Silence is hard to parse

Bug#1015887: debian-installer: Adding https repo doesn't work without manually installing ca-certificates

[Richard Hector]

Package: debian-installer
Severity: important

Dear Maintainer,

Using netinst bullseye 11.4 installer:

https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.4.0-amd64-netinst.iso

I chose to add a network mirror, using https, and the default
'deb.debian.org'.

I used (non-graphical) Expert Mode.

The problem first showed up when tasksel only displayed 'standard system
utilities'. When I went ahead with that, the next screen was a red
'Installation step failed' screen.

The log on tty4 showed various dependency problems.

I tried to 'chroot /target' and 'apt update', which showed certificate
problems. I then ran 'apt install ca-certificates', which worked
(installing from the cd image?), after which 'apt update' worked, and I
was also able to continue successfully with the installer.

I was able to reproduce this in a (kvm/qemu) VM (which is where I
confirmed my steps); the original problem was on an HP Thin Client
(t520). In both cases only 8G of storage was available.

It all works fine using http for the mirror.

I'm happy to do further testing with the VM; the thin client is less
convenient as it has a job to do.

-- System Information:
Debian Release: 11.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-16-amd64 (SMP w/2 CPU threads)
Locale: LANG=en_NZ.UTF-8, LC_CTYPE=en_NZ.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_NZ:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled