Re: SSLUseStapling: ssl handshake fails until httpd restart
On 4 Oct 2015, at 11:38, Kaspar Brand wrote: > > As far as the mod_ssl side is related, it seems to me that for the > "SSLStaplingReturnResponderErrors off" case, we should make sure that we only > staple responses with status "good" (i.e. OCSP_RESPONSE_STATUS_SUCCESSFUL and > V_OCSP_CERTSTATUS_GOOD for the cert). If the OCSP response is successful but the status isn't V_OCSP_CERTSTATUS_GOOD, I'd want httpd to at least log a warning (as well as not stapling the OCSP information). Maybe even add a Warning: header for any client that's interested. I can attempt a patch for this if other people think it'd be useful. -- Tim Bannister – is...@c8h10n4o2.org.uk
Re: SSLUseStapling: ssl handshake fails until httpd restart
On 03.10.2015 12:07, Reindl Harald wrote: > Am 03.10.2015 um 11:16 schrieb Kaspar Brand: >> What do you have security.OCSP.require set to? If it's "true" (a setting >> no longer configurable through the UI, BTW, see >> https://bugzilla.mozilla.org/show_bug.cgi?id=1034360), then Firefox >> shows a fairly peculiar behavior: even when OCSP checking is disabled >> (by setting security.OCSP.enabled to "0", through the "Query OCSP >> responder servers to confirm the current validity of certificates" >> preference in the UI under Advanced -> Certificates), it still includes >> a status_request extension in the TLS handshake, and will subsequently >> fail when it receives a stapled tryLater OCSP response, as long as >> security.OCSP.require=true > > security.OCSP.require=false is the default Oh, looks like Firefox does sort of a "hard fail" when receiving a stapled tryLater response - not what I would expect when security.OCSP.require=false (and also security.OCSP.enabled=0), but apparently, the security.ssl.enable_ocsp_stapling pref has precedence over those. As far as the mod_ssl side is related, it seems to me that for the "SSLStaplingReturnResponderErrors off" case, we should make sure that we only staple responses with status "good" (i.e. OCSP_RESPONSE_STATUS_SUCCESSFUL and V_OCSP_CERTSTATUS_GOOD for the cert). > but it's not only my client with random failed connections This is really due to a very weird effect with ocsp.godaddy.com, as further examination shows. GoDaddy is making use of Akamai's Global Traffic Management service for "load balancing", i.e., depending on where the client is located, you're getting one of the following addresses back when querying for ocsp.godaddy.com.akadns.net (the CNAME for ocsp.godaddy.com): 72.167.18.239 50.63.243.230 72.167.239.239 188.121.36.239 (the list might be incomplete, but it illustrates Akamai's GTM mechanism) For Central Europe, you get back 188.121.36.239 (from GoDaddy's Amsterdam data center), and from what I can tell, this site has a pretty quirky behavior when handling new connections on port 80: the first connection is instantaneously torn down with a TCP reset... but if you immediately try again, the OCSP request is properly handled (for your *.thelounge.net cert e.g., try [1]). Then, after some idle timeout, the same effect kicks in again. I haven't been able to determine how long the timeout is exactly, but judging from your log extracts, it seems to be less than 30 minutes). Strangely enough, the other (US-based) sites don't seem to exhibit this effect. Kaspar [1] http://188.121.36.239/MEgwRjBEMEIwQDAJBgUrDgMCGgUABBS2CA1fbGt26xPkOKX4ZguoUjM0TgQUQMK9J47MNIMwojPX+2yz8LQsgM4CByr+BC4coss=
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 03.10.2015 um 11:16 schrieb Kaspar Brand: On 01.10.2015 16:32, Reindl Harald wrote: Am 01.10.2015 um 16:29 schrieb Plüm, Rüdiger, Vodafone Group: The question is: What happens on Firefox side. Of course it still tries to get to the OCSP server, but it should not cause an error on Firefox side if this does not work. no, it does not because "security.OCSP.enabled = 0" and i saw at least two requests to different servers failing with my Firefox with the responder error from the webserver What do you have security.OCSP.require set to? If it's "true" (a setting no longer configurable through the UI, BTW, see https://bugzilla.mozilla.org/show_bug.cgi?id=1034360), then Firefox shows a fairly peculiar behavior: even when OCSP checking is disabled (by setting security.OCSP.enabled to "0", through the "Query OCSP responder servers to confirm the current validity of certificates" preference in the UI under Advanced -> Certificates), it still includes a status_request extension in the TLS handshake, and will subsequently fail when it receives a stapled tryLater OCSP response, as long as security.OCSP.require=true security.OCSP.require=false is the default but it's not only my client with random failed connections [Sat Oct 03 00:15:01.478741 2015] [ssl:error] [pid 27458] (104)Connection reset by peer: [client 81.223.20.5:59844] AH01977: failed reading line from OCSP server [Sat Oct 03 00:45:01.618792 2015] [ssl:error] [pid 4965] (104)Connection reset by peer: [client 81.223.20.5:33566] AH01977: failed reading line from OCSP server [Sat Oct 03 01:15:01.589643 2015] [ssl:error] [pid 5599] (104)Connection reset by peer: [client 81.223.20.5:36218] AH01977: failed reading line from OCSP server [Sat Oct 03 01:45:01.816132 2015] [ssl:error] [pid 4965] (104)Connection reset by peer: [client 81.223.20.5:38762] AH01977: failed reading line from OCSP server [Sat Oct 03 02:15:01.187187 2015] [ssl:error] [pid 14361] (104)Connection reset by peer: [client 81.223.20.5:41043] AH01977: failed reading line from OCSP server [Sat Oct 03 02:45:01.292205 2015] [ssl:error] [pid 14366] (104)Connection reset by peer: [client 81.223.20.5:42999] AH01977: failed reading line from OCSP server [Sat Oct 03 03:15:01.353077 2015] [ssl:error] [pid 14364] (104)Connection reset by peer: [client 81.223.20.5:45180] AH01977: failed reading line from OCSP server [Sat Oct 03 03:45:01.428090 2015] [ssl:error] [pid 14364] (104)Connection reset by peer: [client 81.223.20.5:46857] AH01977: failed reading line from OCSP server [Sat Oct 03 04:15:02.019261 2015] [ssl:error] [pid 26399] (104)Connection reset by peer: [client 81.223.20.5:49007] AH01977: failed reading line from OCSP server signature.asc Description: OpenPGP digital signature
Re: SSLUseStapling: ssl handshake fails until httpd restart
On 01.10.2015 16:32, Reindl Harald wrote: > Am 01.10.2015 um 16:29 schrieb Plüm, Rüdiger, Vodafone Group: >> The question is: What happens on Firefox side. Of course it still tries to >> get to the OCSP server, but it should not cause an error on Firefox side if >> this does not work. > > no, it does not because "security.OCSP.enabled = 0" and i saw at least > two requests to different servers failing with my Firefox with the > responder error from the webserver What do you have security.OCSP.require set to? If it's "true" (a setting no longer configurable through the UI, BTW, see https://bugzilla.mozilla.org/show_bug.cgi?id=1034360), then Firefox shows a fairly peculiar behavior: even when OCSP checking is disabled (by setting security.OCSP.enabled to "0", through the "Query OCSP responder servers to confirm the current validity of certificates" preference in the UI under Advanced -> Certificates), it still includes a status_request extension in the TLS handshake, and will subsequently fail when it receives a stapled tryLater OCSP response, as long as security.OCSP.require=true. Kaspar
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 01.10.2015 um 16:29 schrieb Plüm, Rüdiger, Vodafone Group: -Ursprüngliche Nachricht- Von: Reindl Harald [mailto:h.rei...@thelounge.net] Gesendet: Donnerstag, 1. Oktober 2015 15:18 An: dev@httpd.apache.org Betreff: Re: SSLUseStapling: ssl handshake fails until httpd restart Am 01.10.2015 um 15:08 schrieb Reindl Harald: Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group: not really, i had the error message just now again in FF, the difference was that now a "try again" loaded the page but with "SSLStaplingReturnResponderErrors" i would expect it invisible to clients in general - GoDaddy seems to have massive problems with their responders the last days and the defaults with stapling enabled make them to a perfect DOS target [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client 10.0.0.99:37860] AH01980: bad response from OCSP server: (none) [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941: stapling_renew_response: responder error SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576) SSLStaplingStandardCacheTimeout 86400 SSLStaplingErrorCacheTimeout 300 SSLStaplingReturnResponderErrors Off What happens if you set SSLStaplingFakeTryLater off in addition? i added that now and will have a look over the serverlogs, it's not happening all the time but very often and so if the logs are clear within 24 hours the problem is likely solved looks not that good - "Connection reset by peer" indicates a failed client request, the other lines could be just internal [Thu Oct 01 15:15:01.495986 2015] [ssl:error] [pid 17468] (104)Connection reset by peer: [client 81.223.20.5:55156] AH01977: failed reading line from OCSP server [Thu Oct 01 15:15:01.496037 2015] [ssl:error] [pid 17468] [client 81.223.20.5:55156] AH01980: bad response from OCSP server: (none) [Thu Oct 01 15:15:01.496057 2015] [ssl:error] [pid 17468] AH01941: stapling_renew_response: responder error The question is: What happens on Firefox side. Of course it still tries to get to the OCSP server, but it should not cause an error on Firefox side if this does not work. no, it does not because "security.OCSP.enabled = 0" and i saw at least two requests to different servers failing with my Firefox with the responder error from the webserver signature.asc Description: OpenPGP digital signature
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 01.10.2015 um 15:08 schrieb Reindl Harald: Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group: not really, i had the error message just now again in FF, the difference was that now a "try again" loaded the page but with "SSLStaplingReturnResponderErrors" i would expect it invisible to clients in general - GoDaddy seems to have massive problems with their responders the last days and the defaults with stapling enabled make them to a perfect DOS target [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client 10.0.0.99:37860] AH01980: bad response from OCSP server: (none) [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941: stapling_renew_response: responder error SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576) SSLStaplingStandardCacheTimeout 86400 SSLStaplingErrorCacheTimeout 300 SSLStaplingReturnResponderErrors Off What happens if you set SSLStaplingFakeTryLater off in addition? i added that now and will have a look over the serverlogs, it's not happening all the time but very often and so if the logs are clear within 24 hours the problem is likely solved looks not that good - "Connection reset by peer" indicates a failed client request, the other lines could be just internal [Thu Oct 01 15:15:01.495986 2015] [ssl:error] [pid 17468] (104)Connection reset by peer: [client 81.223.20.5:55156] AH01977: failed reading line from OCSP server [Thu Oct 01 15:15:01.496037 2015] [ssl:error] [pid 17468] [client 81.223.20.5:55156] AH01980: bad response from OCSP server: (none) [Thu Oct 01 15:15:01.496057 2015] [ssl:error] [pid 17468] AH01941: stapling_renew_response: responder error signature.asc Description: OpenPGP digital signature
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group: -Ursprüngliche Nachricht- Von: Reindl Harald [mailto:h.rei...@thelounge.net] The default for SSLStaplingReturnResponderErrors is relatively odd. Not sure why it has always defaulted to "on" (r829619), but setting it to off should save you further troubles with Firefox clients. not really, i had the error message just now again in FF, the difference was that now a "try again" loaded the page but with "SSLStaplingReturnResponderErrors" i would expect it invisible to clients in general - GoDaddy seems to have massive problems with their responders the last days and the defaults with stapling enabled make them to a perfect DOS target [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client 10.0.0.99:37860] AH01980: bad response from OCSP server: (none) [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941: stapling_renew_response: responder error SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576) SSLStaplingStandardCacheTimeout 86400 SSLStaplingErrorCacheTimeout 300 SSLStaplingReturnResponderErrors Off What happens if you set SSLStaplingFakeTryLater off in addition? i added that now and will have a look over the serverlogs, it's not happening all the time but very often and so if the logs are clear within 24 hours the problem is likely solved thanks! signature.asc Description: OpenPGP digital signature
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 30.09.2015 um 08:42 schrieb Kaspar Brand: On 29.09.2015 18:24, Reindl Harald wrote: i just restarted the servers and disabled stapling since all our servcies where unreachable (before i write the second mail 5 different hosts with several sites where affected) in fact the error caching does more harm than benefits - IHMO a better "could not reach OCSP server or received a error from it" caching would be just temporary disable stapling for 10 minutes instead lead in connections fail even from clients which have disabled OCSP completly firefox refused to open our adminpanel with the error below until i restarted httpd The default for SSLStaplingReturnResponderErrors is relatively odd. Not sure why it has always defaulted to "on" (r829619), but setting it to off should save you further troubles with Firefox clients. not really, i had the error message just now again in FF, the difference was that now a "try again" loaded the page but with "SSLStaplingReturnResponderErrors" i would expect it invisible to clients in general - GoDaddy seems to have massive problems with their responders the last days and the defaults with stapling enabled make them to a perfect DOS target [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client 10.0.0.99:37860] AH01980: bad response from OCSP server: (none) [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941: stapling_renew_response: responder error SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576) SSLStaplingStandardCacheTimeout 86400 SSLStaplingErrorCacheTimeout 300 SSLStaplingReturnResponderErrors Off signature.asc Description: OpenPGP digital signature
Re: SSLUseStapling: ssl handshake fails until httpd restart
On 29.09.2015 18:24, Reindl Harald wrote: > i just restarted the servers and disabled stapling since all our > servcies where unreachable (before i write the second mail 5 different > hosts with several sites where affected) > > in fact the error caching does more harm than benefits - IHMO a better > "could not reach OCSP server or received a error from it" caching would > be just temporary disable stapling for 10 minutes instead lead in > connections fail even from clients which have disabled OCSP completly > >>> firefox refused to open our adminpanel with the error below until i >>> restarted httpd The default for SSLStaplingReturnResponderErrors is relatively odd. Not sure why it has always defaulted to "on" (r829619), but setting it to off should save you further troubles with Firefox clients. Kaspar
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 29.09.2015 um 17:31 schrieb Jeff Trawick: On 09/29/2015 04:20 AM, Reindl Harald wrote: is that by intention? The default timeout before retrying an error seems to be 10 minutes (see http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslstaplingerrorcachetimeout), which is pretty excessive. As far as you recall about the time period before you gave up, was that within 10 minutes? i just restarted the servers and disabled stapling since all our servcies where unreachable (before i write the second mail 5 different hosts with several sites where affected) in fact the error caching does more harm than benefits - IHMO a better "could not reach OCSP server or received a error from it" caching would be just temporary disable stapling for 10 minutes instead lead in connections fail even from clients which have disabled OCSP completly firefox refused to open our adminpanel with the error below until i restarted httpd - i suggest the server should retry SSLUseStapling when a new client connects and it has failed for whatever reason SSLUseStapling On An error occurred during a connection to ***:8443. The OCSP server suggests trying again later. (Error code: sec_error_ocsp_try_server_later) signature.asc Description: OpenPGP digital signature
Re: SSLUseStapling: ssl handshake fails until httpd restart
On 09/29/2015 04:20 AM, Reindl Harald wrote: is that by intention? The default timeout before retrying an error seems to be 10 minutes (see http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslstaplingerrorcachetimeout), which is pretty excessive. As far as you recall about the time period before you gave up, was that within 10 minutes? firefox refused to open our adminpanel with the error below until i restarted httpd - i suggest the server should retry SSLUseStapling when a new client connects and it has failed for whatever reason SSLUseStapling On An error occurred during a connection to ***:8443. The OCSP server suggests trying again later. (Error code: sec_error_ocsp_try_server_later)
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 29.09.2015 um 10:20 schrieb Reindl Harald: is that by intention? firefox refused to open our adminpanel with the error below until i restarted httpd - i suggest the server should retry SSLUseStapling when a new client connects and it has failed for whatever reason SSLUseStapling On An error occurred during a connection to ***:8443. The OCSP server suggests trying again later. (Error code: sec_error_ocsp_try_server_later) as long httpd behaves that way i suggest to re-consider if it's a good idea to use "SSLUseStapling On" as future default - i just disabled it everywhere because *any* of our https-sites using the same GoDaddy wildcard certificate where broken repeatly today with the same error message and after a "apachectl graceful" started to work again signature.asc Description: OpenPGP digital signature
