Bug#933129: apache2: OCSP stapling poorly handled, yielding trylater errors in the client

[Vincent Lefevre]

The upstream bugs to watch for:

  https://bz.apache.org/bugzilla/show_bug.cgi?id=57121
  "ocsp stapling should not pass temporary server outages to clients"

  https://bz.apache.org/bugzilla/show_bug.cgi?id=60182
  "SSLStaplingFakeTryLater Deviates From Documented Behavior of Only
  Being Effective When SSLStaplingReturnResponderErrors is On"

A possible workaround:

  
https://community.letsencrypt.org/t/robust-ocsp-stapling-with-apache-httpd/87896

And the mod_md workaround suggested in the upstream bugs is currently
not possible in Debian as this module is too old for OCSP stapling:

  
https://serverfault.com/questions/1007247/apache-httpd-how-to-enable-ocsp-stapling-with-mod-md

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#933129: apache2: OCSP stapling poorly handled, yielding trylater errors in the client

[Vincent Lefevre]

I eventually had to disable OCSP stapling on my server: errors occur
too frequently, even just after restarting apache.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#933129: apache2: OCSP stapling poorly handled, yielding trylater errors in the client

[Vincent Lefevre]

On 2019-09-26 23:40:45 +0200, Vincent Lefevre wrote:
> Control: found -1 2.4.38-3+deb10u1
> 
> On 2019-07-26 22:30:00 +0200, Vincent Lefevre wrote:
> > I sometimes get SEC_ERROR_OCSP_TRY_SERVER_LATER errors in Firefox
> > when I connect to my web server. The apache log shows errors like
> > 
> > [Fri Jul 26 20:01:31.355081 2019] [ssl:error] [pid 13552:tid 
> > 139871725876992] [client 207.46.13.73:1928] AH02321: empty response from 
> > OCSP server
> > [Fri Jul 26 20:01:31.366890 2019] [ssl:error] [pid 13552:tid 
> > 139871725876992] [client 207.46.13.73:1928] AH01980: bad response from OCSP 
> > server: (none)
> > [Fri Jul 26 20:01:31.366961 2019] [ssl:error] [pid 13552:tid 
> > 139871725876992] AH01941: stapling_renew_response: responder error
> 
> This still occurs. And when it does, I need to restart apache2.

This may be one of the following upstream bugs:

  https://bz.apache.org/bugzilla/show_bug.cgi?id=57121
  "ocsp stapling should not pass temporary server outages to clients"

  https://bz.apache.org/bugzilla/show_bug.cgi?id=61453
  "OCSP Stapling: SSLStaplingFakeTryLater responses cached too long"

  https://bz.apache.org/bugzilla/show_bug.cgi?id=61531
  "SSLStaplingReturnResponderErrors should return last cached response
  if is an error upstream"

The second one has a link to a very simple patch, in case this is
related.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#933129: apache2: OCSP stapling poorly handled, yielding trylater errors in the client

[Vincent Lefevre]

Control: found -1 2.4.38-3+deb10u1

On 2019-07-26 22:30:00 +0200, Vincent Lefevre wrote:
> I sometimes get SEC_ERROR_OCSP_TRY_SERVER_LATER errors in Firefox
> when I connect to my web server. The apache log shows errors like
> 
> [Fri Jul 26 20:01:31.355081 2019] [ssl:error] [pid 13552:tid 139871725876992] 
> [client 207.46.13.73:1928] AH02321: empty response from OCSP server
> [Fri Jul 26 20:01:31.366890 2019] [ssl:error] [pid 13552:tid 139871725876992] 
> [client 207.46.13.73:1928] AH01980: bad response from OCSP server: (none)
> [Fri Jul 26 20:01:31.366961 2019] [ssl:error] [pid 13552:tid 139871725876992] 
> AH01941: stapling_renew_response: responder error

This still occurs. And when it does, I need to restart apache2.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#933129: apache2: OCSP stapling poorly handled, yielding trylater errors in the client

[Vincent Lefevre]

Package: apache2
Version: 2.4.25-3+deb9u7
Severity: important

I sometimes get SEC_ERROR_OCSP_TRY_SERVER_LATER errors in Firefox
when I connect to my web server. The apache log shows errors like

[Fri Jul 26 20:01:31.355081 2019] [ssl:error] [pid 13552:tid 139871725876992] 
[client 207.46.13.73:1928] AH02321: empty response from OCSP server
[Fri Jul 26 20:01:31.366890 2019] [ssl:error] [pid 13552:tid 139871725876992] 
[client 207.46.13.73:1928] AH01980: bad response from OCSP server: (none)
[Fri Jul 26 20:01:31.366961 2019] [ssl:error] [pid 13552:tid 139871725876992] 
AH01941: stapling_renew_response: responder error

I am not the only one getting such an issue. See:

  https://lafibre.info/cryptographie/ocsp-stapling-lets-encrypt/
  
https://community.letsencrypt.org/t/ocsp-error-is-taking-down-my-site-in-firefox/19496/4
  
https://www.reddit.com/r/sysadmin/comments/bh85ze/sectigos_ocsp_stapling_issues_earlier_this_week/

I think that a way to solve this issue is to add another option
specifying how long a cached response remains valid in case of error
(as long as it satisfies the SSLStaplingResponseMaxAge condition).

-- Package-specific info:

-- System Information:
Debian Release: 9.9
  APT prefers oldstable-updates
  APT policy: (500, 'oldstable-updates'), (500, 'oldstable-debug'), (500, 
'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-9-amd64 (SMP w/1 CPU core)
Locale: LANG=POSIX, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=POSIX 
(charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apache2 depends on:
ii  apache2-bin  2.4.25-3+deb9u7
ii  apache2-data 2.4.25-3+deb9u7
ii  apache2-utils2.4.25-3+deb9u7
ii  dpkg 1.18.25
ii  init-system-helpers  1.48
ii  lsb-base 9.20161125
ii  mime-support 3.60
ii  perl 5.24.1-3+deb9u5
ii  procps   2:3.3.12-3+deb9u1

Versions of packages apache2 recommends:
ii  ssl-cert  1.0.39

Versions of packages apache2 suggests:
pn  apache2-doc  
pn  apache2-suexec-pristine | apache2-suexec-custom  
ii  lynx [www-browser]   2.8.9dev11-1

Versions of packages apache2-bin depends on:
ii  libapr1  1.5.2-5
ii  libaprutil1  1.5.4-3
ii  libaprutil1-dbd-sqlite3  1.5.4-3
ii  libaprutil1-ldap 1.5.4-3
ii  libc62.24-11+deb9u4
ii  libldap-2.4-22.4.44+dfsg-5+deb9u2
ii  liblua5.2-0  5.2.4-1.1+b2
ii  libnghttp2-141.18.1-1
ii  libpcre3 2:8.39-3
ii  libssl1.0.2  1.0.2s-1~deb9u1
ii  libxml2  2.9.4+dfsg1-2.2+deb9u2
ii  perl 5.24.1-3+deb9u5
ii  zlib1g   1:1.2.8.dfsg-5

Versions of packages apache2-bin suggests:
pn  apache2-doc  
pn  apache2-suexec-pristine | apache2-suexec-custom  
ii  lynx [www-browser]   2.8.9dev11-1

Versions of packages apache2 is related to:
ii  apache2  2.4.25-3+deb9u7
ii  apache2-bin  2.4.25-3+deb9u7

-- Configuration Files:
/etc/apache2/envvars changed:
unset HOME
if [ "${APACHE_CONFDIR##/etc/apache2-}" != "${APACHE_CONFDIR}" ] ; then
SUFFIX="-${APACHE_CONFDIR##/etc/apache2-}"
else
SUFFIX=
fi
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2$SUFFIX/apache2.pid
export APACHE_RUN_DIR=/var/run/apache2$SUFFIX
export APACHE_LOCK_DIR=/var/lock/apache2$SUFFIX
export APACHE_LOG_DIR=/var/log/apache2$SUFFIX
export LANG=C
export LANG
export MY_IPS="127 ::1/128 155.133.131.76 2001:4b99:1:3:216:3eff:fe20:ac98 
86.75.119.128 2a02:8429:80cd:3100::/56 80.65.226.245"

/etc/apache2/mods-available/ssl.conf changed:

# Pseudo Random Number Generator (PRNG):
# Configure one or more sources to seed the PRNG of the SSL library.
# The seed data should be of good random quality.
# WARNING! On some platforms /dev/random blocks if not enough entropy
# is available. This means you then cannot use the /dev/random device
# because it would lead to very long connection times (as long as
# it requires to make more entropy available). But usually those
# platforms additionally provide a /dev/urandom device which doesn't
# block. So, if available, use this one instead. Read the mod_ssl User
# Manual for more details.
#
SSLRandomSeed startup builtin
SSLRandomSeed startup file:/dev/urandom 512
SSLRandomSeed connect builtin
SSLRandomSeed connect file:/dev/urandom 512
##
##  SSL Global Context
##
##  All SSL configuration in this context applies both to
##  the main server and all SSL-enabled virtual hosts.
##
#
#   Some MIME-types for downloading Certificates and CRLs
#
AddType