Bug#944124: apt fails to verify certificate when using https and ocsp stapling
On 2019-11-09 David Kalnischkies wrote: > On Mon, Nov 04, 2019 at 05:49:53PM +0100, Robert Senger wrote: >> We are running several debian repositories for custom kernel and >> patched deb packages. We use apache2 on Buster, with https enabled, >> to serve the repos. >> >> This worked fine, until we decided to enable ocsp stapling in >> apache2, which runs other vhosts besides the repos. >> >> Since then, apt fails to validate the server's certificate. Error >> message is: >> >> Fehl:15 https://microscopium.de/repos/apt/debian/common buster/patched >> Release >> Certificate verification failed: The certificate is NOT trusted. >> The received OCSP status response is invalid. Could not >> handshake: Error in the certificate verification. [IP: >> fd10:2842:f0d1:101:222:4dff:feb8:17c 8000] >> >> Restarting apache2 helps for a while (apt works, at least once), but >> the error comes up again when apt is run later. > The (part of the) error message is from libgnutls30: > | The certificate is NOT trusted. The received OCSP status response is > invalid. > and is the stringyified GNUTLS_E_CERTIFICATE_VERIFICATION_ERROR error returned > by gnutls_handshake. Hello, currently gnutls-cli connects successfully. I think it is very fishy that restarting the server lets apt succeed for some time. > apts https method is a thin gnutls-powered wrapper around our http method, the > code lives in the TlsFd struct and the UnwrapTLS method, both here: > https://salsa.debian.org/apt-team/apt/blob/master/methods/connect.cc#L802 > As you can see, we have no code dealing with OCSP directly and we do > not keep state ourselves by caching responses or some such, so I am > completely at a lose what apt could be doing wrong here or how its > behaviour would change over multiple runs. > So it is either a bug in gnutls or in how we use it; in both cases I hope the > gnutls maintainers can shine some light on the issue better than I could and > hence I reassign to them. GnuTLS is supposed to handle OCSP stapling transparently, without special client code. https://gnutls.org/manual/gnutls.html#index-OCSP-stapling | Since GnuTLS 3.5.1 the client certificate verification will consider | the [RFC7633] OCSP-Must-staple certificate extension, and will consider | it while checking for stapled OCSP responses. If the extension is | present and no OCSP staple is found, the certificate verification will | fail and the status code GNUTLS_CERT_MISSING_OCSP_STATUS will returned | from the verification function. The apt code seems to be basically ex-client-x509.c https://gnutls.org/manual/gnutls.html#Client-example-with-X_002e509-certificate-support however ex-client-x509.c can connect to / verify the example host. I have not tried apt. But if it really does not succeed finding the difference to ex-client-x509.c should help. >> All web tools tell us that certificate installation and ocsp >> stapling are correct. No other problems with other https clients >> have been observed so far. [...] > It might also help if you could provide a public testcase as [...] AFAIUI microscopium.de is a public testcase. cu Andreas -- `What a good friend you are to him, Dr. Maturin. His other friends are so grateful to you.' `I sew his ears on from time to time, sure'
Bug#944124: apt fails to verify certificate when using https and ocsp stapling
Hi again, On Sat, Nov 09, 2019 at 11:26:59AM +0100, David Kalnischkies wrote: > […] Note that my mail was rejected by the submitters mail server: | : | 88.217.187.146 does not like recipient. | Remote host said: 554 5.7.1 : Recipient address rejected: Access denied | Giving up on 88.217.187.146. I am not sure how actionable this bugreport is if the submitter isn't reachable… Best regards David Kalnischkies signature.asc Description: PGP signature
Bug#944124: apt fails to verify certificate when using https and ocsp stapling
Control: reassign -1 libgnutls30 3.6.10-4 Hi, On Mon, Nov 04, 2019 at 05:49:53PM +0100, Robert Senger wrote: > We are running several debian repositories for custom kernel and patched deb > packages. We use apache2 on Buster, with https enabled, to serve the repos. > > This worked fine, until we decided to enable ocsp stapling in apache2, which > runs other vhosts besides the repos. > > Since then, apt fails to validate the server's certificate. Error message is: > > Fehl:15 https://microscopium.de/repos/apt/debian/common buster/patched Release > Certificate verification failed: The certificate is NOT trusted. The > received > OCSP status response is invalid. Could not handshake: Error in the > certificate > verification. [IP: fd10:2842:f0d1:101:222:4dff:feb8:17c 8000] > > Restarting apache2 helps for a while (apt works, at least once), but the error > comes up again when apt is run later. The (part of the) error message is from libgnutls30: | The certificate is NOT trusted. The received OCSP status response is invalid. and is the stringyified GNUTLS_E_CERTIFICATE_VERIFICATION_ERROR error returned by gnutls_handshake. apts https method is a thin gnutls-powered wrapper around our http method, the code lives in the TlsFd struct and the UnwrapTLS method, both here: https://salsa.debian.org/apt-team/apt/blob/master/methods/connect.cc#L802 As you can see, we have no code dealing with OCSP directly and we do not keep state ourselves by caching responses or some such, so I am completely at a lose what apt could be doing wrong here or how its behaviour would change over multiple runs. So it is either a bug in gnutls or in how we use it; in both cases I hope the gnutls maintainers can shine some light on the issue better than I could and hence I reassign to them. > All web tools tell us that certificate installation and ocsp stapling are > correct. No other problems with other https clients have been observed so far. You might want to retry with gnutls based clients. Many popular https clients are based on openssl, so your testing might not be as diverse as you have thought. It might also help if you could provide a public testcase as – as you might predict – https servers with client certificate setup and stapling aren't exactly common or easy to setup for a quick test; in all honest, I haven't tried in the hope that someone closer to the material can/will. Best regards David Kalnischkies signature.asc Description: PGP signature
Bug#944124: apt fails to verify certificate when using https and ocsp stapling
Package: apt Version: 1.8.4 Severity: normal Dear Maintainer, We are running several debian repositories for custom kernel and patched deb packages. We use apache2 on Buster, with https enabled, to serve the repos. This worked fine, until we decided to enable ocsp stapling in apache2, which runs other vhosts besides the repos. Since then, apt fails to validate the server's certificate. Error message is: Fehl:15 https://microscopium.de/repos/apt/debian/common buster/patched Release Certificate verification failed: The certificate is NOT trusted. The received OCSP status response is invalid. Could not handshake: Error in the certificate verification. [IP: fd10:2842:f0d1:101:222:4dff:feb8:17c 8000] Restarting apache2 helps for a while (apt works, at least once), but the error comes up again when apt is run later. All web tools tell us that certificate installation and ocsp stapling are correct. No other problems with other https clients have been observed so far. Our configuration uses squid-deb-proxy, but disabling the proxy does not resolve the certificate problem. -- Package-specific info: -- apt-config dump -- APT ""; APT::Architecture "amd64"; APT::Build-Essential ""; APT::Build-Essential:: "build-essential"; APT::Install-Recommends "true"; APT::Install-Suggests "0"; APT::Sandbox ""; APT::Sandbox::User "_apt"; APT::Authentication ""; APT::Authentication::TrustCDROM "true"; APT::NeverAutoRemove ""; APT::NeverAutoRemove:: "^firmware-linux.*"; APT::NeverAutoRemove:: "^linux-firmware$"; APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$"; APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$"; APT::NeverAutoRemove:: "^linux-image-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-image-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-headers-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-headers-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-image-extra-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-image-extra-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-modules-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-modules-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-modules-extra-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-modules-extra-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-signed-image-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-signed-image-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-image-unsigned-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-image-unsigned-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^kfreebsd-image-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^kfreebsd-image-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^kfreebsd-headers-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^kfreebsd-headers-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^gnumach-image-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^gnumach-image-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^.*-modules-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^.*-modules-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^.*-kernel-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^.*-kernel-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-backports-modules-.*-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-backports-modules-.*-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-modules-.*-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-modules-.*-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-tools-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-tools-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-cloud-tools-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-cloud-tools-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-buildinfo-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-buildinfo-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^linux-source-5\.2\.0-2-amd64$"; APT::NeverAutoRemove:: "^linux-source-5\.2\.0-3-amd64$"; APT::NeverAutoRemove:: "^postgresql-"; APT::VersionedKernelPackages ""; APT::VersionedKernelPackages:: "linux-image"; APT::VersionedKernelPackages:: "linux-headers"; APT::VersionedKernelPackages:: "linux-image-extra"; APT::VersionedKernelPackages:: "linux-modules"; APT::VersionedKernelPackages:: "linux-modules-extra"; APT::VersionedKernelPackages:: "linux-signed-image"; APT::VersionedKernelPackages:: "linux-image-unsigned"; APT::VersionedKernelPackages:: "kfreebsd-image"; APT::VersionedKernelPackages:: "kfreebsd-headers"; APT::VersionedKernelPackages:: "gnumach-image"; APT::VersionedKernelPackages:: ".*-modules"; APT::VersionedKernelPackages:: ".*-kernel"; APT::VersionedKernelPackages:: "linux-backports-modules-.*"; APT::VersionedKernelPackages:: "linux-modules-.*"; APT::VersionedKernelPackages:: "linux-tools"; APT::VersionedKernelPackages:: "linux-cloud-tools"; APT::VersionedKernelPackages:: "linux-buildinfo"; APT::VersionedKernelPackages:: "linux-source"; APT::Never-MarkAuto-Sections ""; APT::Never-MarkAuto-Sections:: "metapackages"; APT::Never-MarkAuto-Sections:: "contrib/metapackages"; APT::Never-MarkAuto-Sections:: "non-free/metapackages"; APT::Never-MarkAuto-Sections::
