Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Greg,

> > How do I get in contact with somebody who can fix the problem in
> > Debian Buser and/ or the official Debian Buster arm32v7 Docker image?
>
> Looks a bit like 
> to me.

Thanks a lot! Yes, this looks a lot like my problem. Also their
workaround of running "c_rehash" works. :)

The only difference I was able to spot is that my build process does
not give the "qemu: Unsupported syscall: 382" error message. It stays
completely silent.

https://gitlab.com/toertel/docker-image-tls-https-broken/-/jobs/534935085#L317

317 Setting up ca-certificates (20190110) ...
318 Updating certificates in /etc/ssl/certs...
319 128 added, 0 removed; done.
320 Setting up libgssapi-krb5-2:armhf (1.17-3) ...
321 Setting up libcurl4:armhf (7.64.0-4+deb10u1) ...
322 Setting up curl (7.64.0-4+deb10u1) ...
323 Processing triggers for libc-bin (2.28-10) ...
324 Processing triggers for ca-certificates (20190110) ...
325 Updating certificates in /etc/ssl/certs...
326 0 added, 0 removed; done.
327 Running hooks in /etc/ca-certificates/update.d...
328 done.

I implemented the workaround and it also works in GitLab CI.

https://gitlab.com/toertel/docker-image-tls-https-broken/pipelines/142869786

Greetings,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Greg Wooledge]

On Tue, May 05, 2020 at 04:33:35PM +0200, Mark Jonas wrote:
> # ls -l /etc/ssl/certs/4a6481c9.0
> ls: cannot access '/etc/ssl/certs/4a6481c9.0': No such file or directory
> 
> What is the difference between the numbered links and the ones with
> human readable names?
> 
> I *think* that the problem is that these numbered links are missing.
> 
> How do I get in contact with somebody who can fix the problem in
> Debian Buser and/ or the official Debian Buster arm32v7 Docker image?

Looks a bit like 
to me.

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

> > Yes, I have my own Dockerfile and I can add to it whatever I want. But
> > "dpkg-reconfigure ca-certificates" asks a lot of questions. And that
> > list from 1 to 128 might eventually change. So I am puzzled how to
> > automate that without human intervention.
>
> dpkg-reconfigure --default-priority ca-certificates

Yes, now it does not ask questions. But the workaround so far is to
run dpkg-reconfigure once to remove all certificates and then once
again to re-add all of them. I do not know what "dpkg-reconfigure
--default-priority ca-certificates" does.

Thanks for your help,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Greg,

> You... *think* it's there?  Why not actually look?
>
> unicorn:~$ ls -l /etc/ssl/certs/4a6481c9.0
> lrwxrwxrwx 1 root root 27 Jul 14  2018 /etc/ssl/certs/4a6481c9.0 -> 
> GlobalSign_Root_CA_-_R2.pem
>
> It takes a few seconds, and then you can remove all doubt.

Correct, the file is not there. But there are a lot of other links in
/etc/ssl/certs. But those links have real names, not just sequences of
numbers.

# ls -l /etc/ssl/certs/4a6481c9.0
ls: cannot access '/etc/ssl/certs/4a6481c9.0': No such file or directory

What is the difference between the numbered links and the ones with
human readable names?

I *think* that the problem is that these numbered links are missing.

How do I get in contact with somebody who can fix the problem in
Debian Buser and/ or the official Debian Buster arm32v7 Docker image?

Regards,
Mark

# ls -l /etc/ssl/certs/
total 580
lrwxrwxrwx 1 root root 48 May  1 13:06  ACCVRAIZ1.pem ->
/usr/share/ca-certificates/mozilla/ACCVRAIZ1.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  AC_RAIZ_FNMT-RCM.pem ->
/usr/share/ca-certificates/mozilla/AC_RAIZ_FNMT-RCM.crt
lrwxrwxrwx 1 root root 69 May  1 13:06
Actalis_Authentication_Root_CA.pem ->
/usr/share/ca-certificates/mozilla/Actalis_Authentication_Root_CA.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  AddTrust_External_Root.pem
-> /usr/share/ca-certificates/mozilla/AddTrust_External_Root.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  AffirmTrust_Commercial.pem
-> /usr/share/ca-certificates/mozilla/AffirmTrust_Commercial.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  AffirmTrust_Networking.pem
-> /usr/share/ca-certificates/mozilla/AffirmTrust_Networking.crt
lrwxrwxrwx 1 root root 58 May  1 13:06  AffirmTrust_Premium.pem ->
/usr/share/ca-certificates/mozilla/AffirmTrust_Premium.crt
lrwxrwxrwx 1 root root 62 May  1 13:06
AffirmTrust_Premium_ECC.pem ->
/usr/share/ca-certificates/mozilla/AffirmTrust_Premium_ECC.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_1.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_1.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_2.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_2.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_3.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_3.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_4.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_4.crt
lrwxrwxrwx 1 root root 60 May  1 13:06  Atos_TrustedRoot_2011.pem
-> /usr/share/ca-certificates/mozilla/Atos_TrustedRoot_2011.crt
lrwxrwxrwx 1 root root 96 May  1 13:06
Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem ->
/usr/share/ca-certificates/mozilla/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.crt
lrwxrwxrwx 1 root root 64 May  1 13:06
Baltimore_CyberTrust_Root.pem ->
/usr/share/ca-certificates/mozilla/Baltimore_CyberTrust_Root.crt
lrwxrwxrwx 1 root root 62 May  1 13:06
Buypass_Class_2_Root_CA.pem ->
/usr/share/ca-certificates/mozilla/Buypass_Class_2_Root_CA.crt
lrwxrwxrwx 1 root root 62 May  1 13:06
Buypass_Class_3_Root_CA.pem ->
/usr/share/ca-certificates/mozilla/Buypass_Class_3_Root_CA.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  CA_Disig_Root_R2.pem ->
/usr/share/ca-certificates/mozilla/CA_Disig_Root_R2.crt
lrwxrwxrwx 1 root root 51 May  1 13:06  CFCA_EV_ROOT.pem ->
/usr/share/ca-certificates/mozilla/CFCA_EV_ROOT.crt
lrwxrwxrwx 1 root root 69 May  1 13:06
COMODO_Certification_Authority.pem ->
/usr/share/ca-certificates/mozilla/COMODO_Certification_Authority.crt
lrwxrwxrwx 1 root root 73 May  1 13:06
COMODO_ECC_Certification_Authority.pem ->
/usr/share/ca-certificates/mozilla/COMODO_ECC_Certification_Authority.crt
lrwxrwxrwx 1 root root 73 May  1 13:06
COMODO_RSA_Certification_Authority.pem ->
/usr/share/ca-certificates/mozilla/COMODO_RSA_Certification_Authority.crt
lrwxrwxrwx 1 root root 47 May  1 13:06  Certigna.pem ->
/usr/share/ca-certificates/mozilla/Certigna.crt
lrwxrwxrwx 1 root root 59 May  1 13:06  Certinomis_-_Root_CA.pem
-> /usr/share/ca-certificates/mozilla/Certinomis_-_Root_CA.crt
lrwxrwxrwx 1 root root 66 May  1 13:06
Certplus_Class_2_Primary_CA.pem ->
/usr/share/ca-certificates/mozilla/Certplus_Class_2_Primary_CA.crt
lrwxrwxrwx 1 root root 64 May  1 13:06
Certum_Trusted_Network_CA.pem ->
/usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA.crt
lrwxrwxrwx 1 root root 66 May  1 13:06
Certum_Trusted_Network_CA_2.pem ->
/usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA_2.crt
lrwxrwxrwx 1 root root 71 May  1 13:06
Chambers_of_Commerce_Root_-_2008.pem ->
/usr/share/ca-certificates/mozilla/Chambers_of_Commerce_Root_-_2008.crt
lrwxrwxrwx 1 root root 63 May  1 13:06
Comodo_AAA_Services_root.pem ->
/usr/share/ca-certificates/mozilla/Comodo_AAA_Services_root.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  Cybertrust_Global_Root.pem
-> 

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Michael,

yes, I also tried "update-ca-certificates" and it doesn't work.

# curl https://www.google.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

# update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

# curl https://www.google.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I suspect that the certificate database looks good enough for
"update-ca-certificates" to skip over it but it is actually somehow
broken.

# ls -la /etc/ssl/certs/
total 592
drwxr-xr-x 1 root root   4096 May  5 13:57  .
drwxr-xr-x 1 root root   4096 May  1 13:06  ..
lrwxrwxrwx 1 root root 48 May  1 13:06  ACCVRAIZ1.pem ->
/usr/share/ca-certificates/mozilla/ACCVRAIZ1.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  AC_RAIZ_FNMT-RCM.pem ->
/usr/share/ca-certificates/mozilla/AC_RAIZ_FNMT-RCM.crt
lrwxrwxrwx 1 root root 69 May  1 13:06
Actalis_Authentication_Root_CA.pem ->
/usr/share/ca-certificates/mozilla/Actalis_Authentication_Root_CA.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  AddTrust_External_Root.pem
-> /usr/share/ca-certificates/mozilla/AddTrust_External_Root.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  AffirmTrust_Commercial.pem
-> /usr/share/ca-certificates/mozilla/AffirmTrust_Commercial.crt
lrwxrwxrwx 1 root root 61 May  1 13:06  AffirmTrust_Networking.pem
-> /usr/share/ca-certificates/mozilla/AffirmTrust_Networking.crt
lrwxrwxrwx 1 root root 58 May  1 13:06  AffirmTrust_Premium.pem ->
/usr/share/ca-certificates/mozilla/AffirmTrust_Premium.crt
lrwxrwxrwx 1 root root 62 May  1 13:06
AffirmTrust_Premium_ECC.pem ->
/usr/share/ca-certificates/mozilla/AffirmTrust_Premium_ECC.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_1.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_1.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_2.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_2.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_3.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_3.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  Amazon_Root_CA_4.pem ->
/usr/share/ca-certificates/mozilla/Amazon_Root_CA_4.crt
lrwxrwxrwx 1 root root 60 May  1 13:06  Atos_TrustedRoot_2011.pem
-> /usr/share/ca-certificates/mozilla/Atos_TrustedRoot_2011.crt
lrwxrwxrwx 1 root root 96 May  1 13:06
Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem ->
/usr/share/ca-certificates/mozilla/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.crt
lrwxrwxrwx 1 root root 64 May  1 13:06
Baltimore_CyberTrust_Root.pem ->
/usr/share/ca-certificates/mozilla/Baltimore_CyberTrust_Root.crt
lrwxrwxrwx 1 root root 62 May  1 13:06
Buypass_Class_2_Root_CA.pem ->
/usr/share/ca-certificates/mozilla/Buypass_Class_2_Root_CA.crt
lrwxrwxrwx 1 root root 62 May  1 13:06
Buypass_Class_3_Root_CA.pem ->
/usr/share/ca-certificates/mozilla/Buypass_Class_3_Root_CA.crt
lrwxrwxrwx 1 root root 55 May  1 13:06  CA_Disig_Root_R2.pem ->
/usr/share/ca-certificates/mozilla/CA_Disig_Root_R2.crt
lrwxrwxrwx 1 root root 51 May  1 13:06  CFCA_EV_ROOT.pem ->
/usr/share/ca-certificates/mozilla/CFCA_EV_ROOT.crt
lrwxrwxrwx 1 root root 69 May  1 13:06
COMODO_Certification_Authority.pem ->
/usr/share/ca-certificates/mozilla/COMODO_Certification_Authority.crt
lrwxrwxrwx 1 root root 73 May  1 13:06
COMODO_ECC_Certification_Authority.pem ->
/usr/share/ca-certificates/mozilla/COMODO_ECC_Certification_Authority.crt
lrwxrwxrwx 1 root root 73 May  1 13:06
COMODO_RSA_Certification_Authority.pem ->
/usr/share/ca-certificates/mozilla/COMODO_RSA_Certification_Authority.crt
lrwxrwxrwx 1 root root 47 May  1 13:06  Certigna.pem ->
/usr/share/ca-certificates/mozilla/Certigna.crt
lrwxrwxrwx 1 root root 59 May  1 13:06  Certinomis_-_Root_CA.pem
-> /usr/share/ca-certificates/mozilla/Certinomis_-_Root_CA.crt
lrwxrwxrwx 1 root root 66 May  1 13:06
Certplus_Class_2_Primary_CA.pem ->
/usr/share/ca-certificates/mozilla/Certplus_Class_2_Primary_CA.crt
lrwxrwxrwx 1 root root 64 May  1 13:06
Certum_Trusted_Network_CA.pem ->
/usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA.crt
lrwxrwxrwx 1 root root 66 May  1 13:06
Certum_Trusted_Network_CA_2.pem ->
/usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA_2.crt
lrwxrwxrwx 1 root root 71 May  1 13:06

Re: armhf: buster: TLS / HTTPS partly broken

[Greg Wooledge]

> > > 1613  stat64("/etc/ssl/certs/4a6481c9.0", 0x7ec95160) = -1 ENOENT (No
> > > such file or directory)
> >
> > Presumably ca-certificates postinst script haven't run, because these
> > symlinks missing ain't normal.
> 
> Ubuntu 18.04 on my PC gives more or less the same errors but succeeds.
> So I have some doubt. I think the symlinks are there.

You... *think* it's there?  Why not actually look?

unicorn:~$ ls -l /etc/ssl/certs/4a6481c9.0
lrwxrwxrwx 1 root root 27 Jul 14  2018 /etc/ssl/certs/4a6481c9.0 -> 
GlobalSign_Root_CA_-_R2.pem

It takes a few seconds, and then you can remove all doubt.

Be sure you issue the command within the chroot/container.

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

On Tue, May 05, 2020 at 08:44:43AM +0200, Mark Jonas wrote:
> Hi Reco,
> 
> > > What now? How do I get this fixed in Debian and/ or the official
> > > container image?
> >
> > I was under the impression that you're creating your own docker
> > container anyway.
> > Add it to docker build file or whatever it's called.
> 
> Yes, I have my own Dockerfile and I can add to it whatever I want. But
> "dpkg-reconfigure ca-certificates" asks a lot of questions. And that
> list from 1 to 128 might eventually change. So I am puzzled how to
> automate that without human intervention.

dpkg-reconfigure --default-priority ca-certificates

> I am also very much interested in getting the attention of the right
> person to fix the official Debian Docker base image. Do you have an
> idea whom I shall contact?

No idea here.

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[Andrei POPESCU]

On Ma, 05 mai 20, 08:44:43, Mark Jonas wrote:
> 
> I am also very much interested in getting the attention of the right
> person to fix the official Debian Docker base image. Do you have an
> idea whom I shall contact?

You didn't mention where you got your images from, but maybe this is a 
start:
https://hub.docker.com/_/debian/

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: armhf: buster: TLS / HTTPS partly broken

[Michael Howard]

On 05/05/2020 07:44, Mark Jonas wrote:

Hi Reco,


What now? How do I get this fixed in Debian and/ or the official
container image?

I was under the impression that you're creating your own docker
container anyway.
Add it to docker build file or whatever it's called.

Yes, I have my own Dockerfile and I can add to it whatever I want. But
"dpkg-reconfigure ca-certificates" asks a lot of questions. And that
list from 1 to 128 might eventually change. So I am puzzled how to
automate that without human intervention.




Does 'update-ca-certificates' not work? Doesn't need interaction. 
Apologies if I missed something, haven't read the whole thread.


--
Michael Howard

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

> > What now? How do I get this fixed in Debian and/ or the official
> > container image?
>
> I was under the impression that you're creating your own docker
> container anyway.
> Add it to docker build file or whatever it's called.

Yes, I have my own Dockerfile and I can add to it whatever I want. But
"dpkg-reconfigure ca-certificates" asks a lot of questions. And that
list from 1 to 128 might eventually change. So I am puzzled how to
automate that without human intervention.

I am also very much interested in getting the attention of the right
person to fix the official Debian Docker base image. Do you have an
idea whom I shall contact?

Greetings,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

Hi.

On Tue, May 05, 2020 at 08:05:04AM +0200, Mark Jonas wrote:
> What now? How do I get this fixed in Debian and/ or the official
> container image?

I was under the impression that you're creating your own docker
container anyway.
Add it to docker build file or whatever it's called.

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

> > 1613  stat64("/etc/ssl/certs/4a6481c9.0", 0x7ec95160) = -1 ENOENT (No
> > such file or directory)
>
> Presumably ca-certificates postinst script haven't run, because these
> symlinks missing ain't normal.

Ubuntu 18.04 on my PC gives more or less the same errors but succeeds.
So I have some doubt. I think the symlinks are there.

openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = 6
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 6
stat("/etc/ssl/certs/99bdd351.0", 0x7ffc3c886370) = -1 ENOENT (No such
file or directory)

> So, start with "dpkg-reconfigure ca-certificates", and if it does not
> fix it - "apt-get install --reinstall ca-certificates".

This does not work either. But the following works. And this is super confusing.

1. Run "dpkg-reconfigure ca-certificates"
   Trust new certificates from certificate authorities? 1 (yes)
   Certificates to activate: (empty list)
   Updating certificates in /etc/ssl/certs...
   0 added, 128 removed; done.

2. Run "dpkg-reconfigure ca-certificates"
   Trust new certificates from certificate authorities? 1 (yes)
   Certificates to activate: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124
125 126 127 128
   Updating certificates in /etc/ssl/certs...
   128 added, 0 removed; done.

3. "curl https://www.google.com;  now succeeds.

What now? How do I get this fixed in Debian and/ or the official
container image?

Is there a way to automate the above so I can do it as a workaround in
the container creation?

Greetings,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

Hi.

On Mon, May 04, 2020 at 08:41:43PM +0200, Mark Jonas wrote:
> Hi Reco,
> 
> > > I used the identical image to run the container on an amhf host
> > > (Raspberry Pi 3). So there is now no QEMU in the way.
> >
> > Curious. Just tested it with curl at Marvell Armada 385 (runs Debian 10,
> > armhf), works as supposed to.
> > I could also test it on Exynos 5422 (also runs Debian 10, armhf), but
> > it'll be the same.
> 
> Do you want to try the Docker image on one of these?

Maybe it'll come to it.

> Maybe the problem is not Debian itself but only the official Debian
> Docker image?

Most probably, see below.

> 1613  stat64("/etc/ssl/certs/4a6481c9.0", 0x7ec95160) = -1 ENOENT (No
> such file or directory)

Presumably ca-certificates postinst script haven't run, because these
symlinks missing ain't normal.
So, start with "dpkg-reconfigure ca-certificates", and if it does not
fix it - "apt-get install --reinstall ca-certificates".

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

> > I used the identical image to run the container on an amhf host
> > (Raspberry Pi 3). So there is now no QEMU in the way.
>
> Curious. Just tested it with curl at Marvell Armada 385 (runs Debian 10,
> armhf), works as supposed to.
> I could also test it on Exynos 5422 (also runs Debian 10, armhf), but
> it'll be the same.

Do you want to try the Docker image on one of these? Maybe the problem
is not Debian itself but only the official Debian Docker image?

> > curl https://www.google.com still fails on the armhf host. So QEMU is
> > out of the game.
>
> Ok. Is it possible to run curl via strace from inside the docker?
> Something like this would be perfect (-o designates an output file):
>
> strace -o /tmp/curl -e trace=file curl https://www.google.com

Please have a look at the reply I send to Tomas. There is the complete
strace output.

> Specifically, it should try to open a symlink to
> /etc/ssl/certs/GlobalSign_Root_CA_-_R2.pem.
> Here it's called /etc/ssl/certs/4a6481c9.0, may be machine-specific.

Yes, it tries to open something like that and fails. But on my PC,
where curl works, the trace shows similar failures.

Raspberry Pi Docker host, armhf Docker container snippet:

1613  openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = 4
1613  stat64("/etc/ssl/certs/99bdd351.0", 0x7ec95160) = -1 ENOENT (No
such file or directory)
1613  openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
1613  stat64("/etc/ssl/certs/4a6481c9.0", 0x7ec95160) = -1 ENOENT (No
such file or directory)
1613  stat64("/etc/ssl/certs/4a6481c9.0", 0x7ec95160) = -1 ENOENT (No
such file or directory)
1613  +++ exited with 60 +++

PC strace snippet:

5524  openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 4
5524  openat(AT_FDCWD, "/dev/random", O_RDONLY) = 5
5524  openat(AT_FDCWD, "/dev/srandom", O_RDONLY) = -1 ENOENT (No such
file or directory)
5524  openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = 6
5524  openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 6
5524  stat("/etc/ssl/certs/99bdd351.0", 0x760b7060) = -1 ENOENT
(No such file or directory)
5524  openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 6
5524  +++ exited with 0 +++

Greetings,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Tomas,

> > Yes, "curl -k https:/www.google.com" succeeds.
>
> Then it's quite probable that the problem lies with certificate
> resolution. Either it doesn't find a trusted root cert to validate
> the server against, or the validation fails.
>
> You might try curl's -v option (with and without -k) to see whether
> it sheds some light.

# curl -v https://www.google.com
* Expire in 0 ms for 6 (transfer 0x109d880)
* Expire in 1 ms for 1 (transfer 0x109d880)
* Expire in 0 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 0 ms for 1 (transfer 0x109d880)
* Expire in 0 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 1 ms for 1 (transfer 0x109d880)
* Expire in 1 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 3 ms for 1 (transfer 0x109d880)
* Expire in 3 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 3 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 5 ms for 1 (transfer 0x109d880)
*   Trying 216.58.207.164...
* TCP_NODELAY set
* Expire in 149991 ms for 3 (transfer 0x109d880)
* Expire in 200 ms for 4 (transfer 0x109d880)
* Connected to www.google.com (216.58.207.164) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

# curl -vk https://www.google.com
* Expire in 0 ms for 6 (transfer 0x133a880)
* Expire in 1 ms for 1 (transfer 0x133a880)
[.. skipping 46 more or less identical lines ..]
* Expire in 4 ms for 1 (transfer 0x133a880)
*   Trying 216.58.207.164...
* TCP_NODELAY set
* Expire in 149993 ms for 3 (transfer 0x133a880)
* Expire in 200 ms for 4 (transfer 0x133a880)
* Connected to www.google.com (216.58.207.164) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC;
CN=www.google.com
*  start date: Apr  7 09:49:21 2020 GMT
*  expire date: Jun 30 09:49:21 2020 GMT
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify result: unable to get local issuer
certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x133a880)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Mon, 04 May 2020 17:57:40 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< set-cookie: 1P_JAR=2020-05-04-17; expires=Wed, 03-Jun-2020 17:57:40
GMT; path=/; domain=.google.com; Secure
< set-cookie: 
NID=203=NJeeaDepuErdSOKYdHIR6vtnByU05gHO2DzxoRS3puHM4AsMlNZ5J2aksbNJrJQxfuGuBx_OaG3uyPuuF5tRqJEa4mGmreZ2F9ilyqksUryBh5z7N5y1_QDbDzCvkme1XonAIo_V7rw99ejIfqk8U1nL_tOw5OUSrBZffdLHchA;
expires=Tue, 03-Nov-2020 17:57:40 GMT; path=/; domain=.google.com;
HttpOnly
< alt-svc: h3-Q050=":443"; ma=2592000,h3-Q049=":443";

Re: armhf: buster: TLS / HTTPS partly broken

[tomas]

On Mon, May 04, 2020 at 02:39:09PM +0200, Mark Jonas wrote:
> Hi Thomas,
> 
> > > curl https://www.google.com still fails on the armhf host. So QEMU is
> > > out of the game.
> >
> > Someone hinted at ca_certificates. To verify that, you could try with
> > the option "-k" for curl. Then the server certificate isn't checked.
> 
> Yes, "curl -k https:/www.google.com" succeeds.

Then it's quite probable that the problem lies with certificate
resolution. Either it doesn't find a trusted root cert to validate
the server against, or the validation fails.

You might try curl's -v option (with and without -k) to see whether
it sheds some light.

Also the proposal brought on this list of looking at strace. Perhaps
limit the trace to file operations, like so:

  strace -f -e trace=%file -o trace.out 

This would let you see where curl is looking for certs files.

Cheers
-- t


signature.asc
Description: Digital signature

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Thomas,

> > curl https://www.google.com still fails on the armhf host. So QEMU is
> > out of the game.
>
> Someone hinted at ca_certificates. To verify that, you could try with
> the option "-k" for curl. Then the server certificate isn't checked.

Yes, "curl -k https:/www.google.com" succeeds.

I am 100% sure that the ca-certificates package is installed.

# dpkg --get-selections | grep ca-certificates
ca-certificatesinstall

> Of course this may be a bad idea for a permanent "solution", but would
> allow you to bisect the problem.

I do not understand how that helps.

In my real use case the Logitech Media Server shows the same problem
as curl. I am just using curl here because it is a Debian supported
package and the problem is way easier to reproduce.

Greetings,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

Hi.

On Mon, May 04, 2020 at 01:49:34PM +0200, Mark Jonas wrote:
> Hi Reco,
> 
> > > > Ok. Can you run tcpdump while you're running curl?
> > > > Specifically,
> > > >
> > > > tcpdump -s0 -pnni any -w /tmp/curl.pcap tcp port 443
> > >
> > > I tried to dump from within the running container but failed.
> >
> > It's way too complicated. Docker is basically a one big NAT, so please
> > run tcpdump on a host instead.
> 
> I used the identical image to run the container on an amhf host
> (Raspberry Pi 3). So there is now no QEMU in the way.

Curious. Just tested it with curl at Marvell Armada 385 (runs Debian 10,
armhf), works as supposed to.
I could also test it on Exynos 5422 (also runs Debian 10, armhf), but
it'll be the same.


> > But this hiccup gave me an idea - maybe libssl on armhf is perfectly
> > fine, but it's qemu which fails to emulate certain CPU instruction.
> 
> curl https://www.google.com still fails on the armhf host. So QEMU is
> out of the game.

Ok. Is it possible to run curl via strace from inside the docker?
Something like this would be perfect (-o designates an output file):

strace -o /tmp/curl -e trace=file curl https://www.google.com


Specifically, it should try to open a symlink to
/etc/ssl/certs/GlobalSign_Root_CA_-_R2.pem.
Here it's called /etc/ssl/certs/4a6481c9.0, may be machine-specific.


> Packet capturing now also worked. For capturing QEMU was the problem.
> I also captured aria2c (succeeds with warning) and wget (silently
> succeeds).

For the archives, curl's pcap looks like this:

SYN
SYN,ACK
TLS client hello
TLS server hello, change cipher spec (an upgrade from TLSv1.2 to TLS1.3)
Alert (Level: Fatal, Description: Unknown CA) <- curl sends this
RST
FIN,ACK

wget and aria show what they're supposed to (TLS handshake, Application
Data ping pong).

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[tomas]

On Mon, May 04, 2020 at 01:49:34PM +0200, Mark Jonas wrote:
> Hi Reco,
> 
> > > > Ok. Can you run tcpdump while you're running curl?
> > > > Specifically,
> > > >
> > > > tcpdump -s0 -pnni any -w /tmp/curl.pcap tcp port 443
> > >
> > > I tried to dump from within the running container but failed.
> >
> > It's way too complicated. Docker is basically a one big NAT, so please
> > run tcpdump on a host instead.
> 
> I used the identical image to run the container on an amhf host
> (Raspberry Pi 3). So there is now no QEMU in the way.
> 
> > But this hiccup gave me an idea - maybe libssl on armhf is perfectly
> > fine, but it's qemu which fails to emulate certain CPU instruction.
> 
> curl https://www.google.com still fails on the armhf host. So QEMU is
> out of the game.

Someone hinted at ca_certificates. To verify that, you could try with
the option "-k" for curl. Then the server certificate isn't checked.

Of course this may be a bad idea for a permanent "solution", but would
allow you to bisect the problem.

Cheers
-- t


signature.asc
Description: Digital signature

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

> > > Ok. Can you run tcpdump while you're running curl?
> > > Specifically,
> > >
> > > tcpdump -s0 -pnni any -w /tmp/curl.pcap tcp port 443
> >
> > I tried to dump from within the running container but failed.
>
> It's way too complicated. Docker is basically a one big NAT, so please
> run tcpdump on a host instead.

I used the identical image to run the container on an amhf host
(Raspberry Pi 3). So there is now no QEMU in the way.

> But this hiccup gave me an idea - maybe libssl on armhf is perfectly
> fine, but it's qemu which fails to emulate certain CPU instruction.

curl https://www.google.com still fails on the armhf host. So QEMU is
out of the game.

Packet capturing now also worked. For capturing QEMU was the problem.
I also captured aria2c (succeeds with warning) and wget (silently
succeeds). You can download the capture files from
https://fil.email/pzzgUgVp . The link is good for one week and is from
filemail.com.

Thanks for your help,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

Hi.

On Mon, May 04, 2020 at 09:27:14AM +0200, Mark Jonas wrote:
> >> >> curl: (60) SSL certificate problem: unable to get local issuer 
> >> >> certificate
> >> >>
> >> >> Does that mean a TLS library does not feature all required protocols on 
> >> >> armhf?
> >> >
> >> > TLS library that curl uses (openssl) is perfectly fine, but it cannot
> >> > validate any certificate unless you provide it with root CA
> >> > certificates.
> >> > So it likely means you haven't installed "ca-certificates" package.
> >>
> >> This is what it looks like. But actually I installed ca-certificates.
> >
> > Ok. Can you run tcpdump while you're running curl?
> > Specifically,
> >
> > tcpdump -s0 -pnni any -w /tmp/curl.pcap tcp port 443
> 
> I tried to dump from within the running container but failed.

It's way too complicated. Docker is basically a one big NAT, so please
run tcpdump on a host instead.

But this hiccup gave me an idea - maybe libssl on armhf is perfectly
fine, but it's qemu which fails to emulate certain CPU instruction.

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

>> >> curl: (60) SSL certificate problem: unable to get local issuer certificate
>> >>
>> >> Does that mean a TLS library does not feature all required protocols on 
>> >> armhf?
>> >
>> > TLS library that curl uses (openssl) is perfectly fine, but it cannot
>> > validate any certificate unless you provide it with root CA
>> > certificates.
>> > So it likely means you haven't installed "ca-certificates" package.
>>
>> This is what it looks like. But actually I installed ca-certificates.
>
> Ok. Can you run tcpdump while you're running curl?
> Specifically,
>
> tcpdump -s0 -pnni any -w /tmp/curl.pcap tcp port 443

I tried to dump from within the running container but failed.

# tcpdump -s0 -pnni any -w /tmp/curl-certificate-problem.pcap tcp port 443
Unsupported setsockopt level=263 optname=8
getsockopt level=263 optname=11 not yet supported
tcpdump: WARNING: can't get TPACKET_V3 header len on packet socket:
Operation not supported
Warning: Kernel filter failed: Bad file descriptor
Unsupported setsockopt level=1 optname=27
tcpdump: can't remove kernel filter: Protocol not available

The container was started as follows on an amd64 host running qemu-arm-static:

$ docker run -it --rm toertel/test-tls-https-broken:arm32v7-buster-latest

I gave it a try with a stripped down command and it did not work either.

# tcpdump -w /tmp/curl-certificate-problem.pcap port 443
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unsupported ioctl: cmd=0x8946
Unsupported ioctl: cmd=0x8946
Unsupported ioctl: cmd=0x8946
Unsupported ioctl: cmd=0x8946
Unsupported ioctl: cmd=0x8946
Unsupported ioctl: cmd=0x8946
Unsupported setsockopt level=263 optname=8
getsockopt level=263 optname=11 not yet supported
tcpdump: Can't open netlink socket 96:Protocol family not supported

Thanks for your help,
Mark

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

On Sun, May 03, 2020 at 07:20:13PM +0200, Mark Jonas wrote:
> Hi Reco,
> 
> >> curl: (60) SSL certificate problem: unable to get local issuer certificate
> >>
> >> Does that mean a TLS library does not feature all required protocols on 
> >> armhf?
> >
> > TLS library that curl uses (openssl) is perfectly fine, but it cannot
> > validate any certificate unless you provide it with root CA
> > certificates.
> > So it likely means you haven't installed "ca-certificates" package.
> 
> This is what it looks like. But actually I installed ca-certificates.

Ok. Can you run tcpdump while you're running curl?
Specifically,

tcpdump -s0 -pnni any -w /tmp/curl.pcap tcp port 443

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi Reco,

>> curl: (60) SSL certificate problem: unable to get local issuer certificate
>>
>> Does that mean a TLS library does not feature all required protocols on 
>> armhf?
>
> TLS library that curl uses (openssl) is perfectly fine, but it cannot
> validate any certificate unless you provide it with root CA
> certificates.
> So it likely means you haven't installed "ca-certificates" package.

This is what it looks like. But actually I installed ca-certificates.

This is an excerpt of the relevant part of the the Dockerfile [1]
where the packages are installed:

RUN apt-get update && \
  apt-get -y --no-install-recommends install \
curl \
ca-certificates \
tzdata \
&& \
  apt-get clean && \
  rm -rf /var/lib/apt/lists/*

I also think that wget would not work or at least give a warning in
case there were no certificates at all.

Last but not least, the identical Dockerfile produces images for amd64
and arm64 where curl and aria2 work without hiccups. And it works
flawlessly on Stretch using the same Dockerfile.

Greetings,
Mark


[1]: 
https://gitlab.com/toertel/docker-image-tls-https-broken/-/blob/master/Dockerfile.j2

Re: armhf: buster: TLS / HTTPS partly broken

[Reco]

Hi.

On Sun, May 03, 2020 at 02:40:14PM +0200, Mark Jonas wrote:
> curl: (60) SSL certificate problem: unable to get local issuer certificate
>
> Does that mean a TLS library does not feature all required protocols on armhf?

TLS library that curl uses (openssl) is perfectly fine, but it cannot
validate any certificate unless you provide it with root CA
certificates.
So it likely means you haven't installed "ca-certificates" package.

Reco

Re: armhf: buster: TLS / HTTPS partly broken

[Andrei POPESCU]

On Du, 03 mai 20, 14:40:14, Mark Jonas wrote:
> 
> Does anybody have an idea what the problem might be? Who can / should
> tackle the problem?
> 
> I did not report the problem using reportbug because I have no clue
> which package is causing the problem.

You could check what curl, aria2 and LMS have in common, but not with 
wget.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

armhf: buster: TLS / HTTPS partly broken

[Mark Jonas]

Hi,

I am building Docker images for amd64, armhf, and arm64. I have a very
simple container based on debian:buster where curl works fine on amd64
and arm64 but fails on armhf [1]. This makes it very easy to reproduce
the problem.

# curl --version
curl 7.64.0 (arm-unknown-linux-gnueabihf) libcurl/7.64.0
OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2
(+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM
NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

# curl https://www.google.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

The error occurs on a real armhf target (Raspberry Pi 3) as well as
with QEMU (tested with
3.1.0-2 and v4.2.0-7).

The error cannot be reproduced with debian:stretch. [2]

The error cannot be reproduced with ubuntu:bionic or ubuntu:focal. [3]

With wget it works fine. None the less, I doubt that curl itself it
the source of the problem. The Logitech Media Server package [4] (not
an official Debian package) shows the problem as well. LMS is written
using Perl (mainly) and does not use curl.

I also gave aria2 a try. It downloads but gives a warning on armhf.

# aria2c https://www.google.com
[..]
05/03 12:32:37 [WARN] aria2c had to connect to the other side using an
unknown TLS protocol. The integrity and confidentiality of the
connection might be compromised.
Peer: www.google.com (216.58.207.164:443)

Does that mean a TLS library does not feature all required protocols on armhf?

Does anybody have an idea what the problem might be? Who can / should
tackle the problem?

I did not report the problem using reportbug because I have no clue
which package is causing the problem.

Greetings,
Mark

[1] https://gitlab.com/toertel/docker-image-tls-https-broken
[2] https://gitlab.com/toertel/docker-image-tls-https-broken/pipelines/141798495
[3] https://gitlab.com/toertel/docker-image-tls-https-broken/pipelines/141820625
[4] http://downloads.slimdevices.com/LogitechMediaServer_v7.9.2/