Re: `ssl_fc_has_early` fetcher and 0rtt
> > That seems strange indeed but looking at the code that's what I'm > > seeing. Was your access to ssl_fc_has_early placed before or after the > > rule above ? If it's after it must indeed report false. fetcher is placed before the rule > > I seem to remember there was one but can't find it, so I may have been > > confused. With this said, it doesn't provide a big information since > > once the handshake is completed, it's exactly identical to a regular > > one. But it can be nice for statistics at least. > > > > Pretty sure the original version always returned 1 if there were early > data, but we changed it so that it would return 0 once the handshake was > done, as we thought it was more useful, that may be what you're thinking > about. Yes I indeed found the commit which changed the behaviour. In fact I probably mixed the true meaning of the definition in the doc and the old blog post about it which is vague. Now everything is clear. >From our point of view, it's interesting to check some behavioural changes we can make on the L4 layer (e.g. hash including source port or not). > That indeed tells me something. However I checked if CO_FL_EARLY_DATA > persists, and apparently we kill it very early once we have everything > so it's not as if we could trivially add a new sample fetch function to > report its past status. > > Also I seem to remember that we concluded that depending on the timing > and how data were aggregated and processed in the SSL lib, we couldn't > reliably report the use of early data if those were converted very early. At some point I was asking myself whether this could be done but simply to give a signal without any guarantee, but you seemed to conclude that it's not reliable enough to become a new fetcher. -- William
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 09, 2020 at 05:43:08PM +0200, Olivier Houchard wrote: > > I seem to remember there was one but can't find it, so I may have been > > confused. With this said, it doesn't provide a big information since > > once the handshake is completed, it's exactly identical to a regular > > one. But it can be nice for statistics at least. > > > > Pretty sure the original version always returned 1 if there were early > data, but we changed it so that it would return 0 once the handshake was > done, as we thought it was more useful, that may be what you're thinking > about. That indeed tells me something. However I checked if CO_FL_EARLY_DATA persists, and apparently we kill it very early once we have everything so it's not as if we could trivially add a new sample fetch function to report its past status. Also I seem to remember that we concluded that depending on the timing and how data were aggregated and processed in the SSL lib, we couldn't reliably report the use of early data if those were converted very early. Willy
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 09, 2020 at 05:35:28PM +0200, Willy Tarreau wrote: > On Wed, Sep 09, 2020 at 04:57:58PM +0200, William Dauchy wrote: > > > I think it's not easy to reproduce these tests, you need a high enough > > > latency between haproxy and the client so that the handshake is not > > > already completed when you evaluate the rule, and of course you need > > > to make sure the client sends using early data. I don't remember how > > > Olivier used to run his tests but I remember that it was a bit tricky, > > > so it's very possible that you never fall into the situation where you > > > can see the unvalidated early data yet. > > > > It means my understanding of this fetcher was wrong indeed. > > For me the protection was here: > > http-request wait-for-handshake if ! METH_GET > > and the fetcher here to log whether it was a 0rtt request or not. > > In reality, it means all our requests have completed the handshake > > when the rule is evaluated (which is surprising looking at the number > > we have). > > That seems strange indeed but looking at the code that's what I'm > seeing. Was your access to ssl_fc_has_early placed before or after the > rule above ? If it's after it must indeed report false. > > > So maybe we can possibly work on an alternative fetcher to know > > whether this was a 0rtt request? Or is there another way? > > I seem to remember there was one but can't find it, so I may have been > confused. With this said, it doesn't provide a big information since > once the handshake is completed, it's exactly identical to a regular > one. But it can be nice for statistics at least. > Pretty sure the original version always returned 1 if there were early data, but we changed it so that it would return 0 once the handshake was done, as we thought it was more useful, that may be what you're thinking about. Olivier
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 09, 2020 at 04:57:58PM +0200, William Dauchy wrote: > > I think it's not easy to reproduce these tests, you need a high enough > > latency between haproxy and the client so that the handshake is not > > already completed when you evaluate the rule, and of course you need > > to make sure the client sends using early data. I don't remember how > > Olivier used to run his tests but I remember that it was a bit tricky, > > so it's very possible that you never fall into the situation where you > > can see the unvalidated early data yet. > > It means my understanding of this fetcher was wrong indeed. > For me the protection was here: > http-request wait-for-handshake if ! METH_GET > and the fetcher here to log whether it was a 0rtt request or not. > In reality, it means all our requests have completed the handshake > when the rule is evaluated (which is surprising looking at the number > we have). That seems strange indeed but looking at the code that's what I'm seeing. Was your access to ssl_fc_has_early placed before or after the rule above ? If it's after it must indeed report false. > So maybe we can possibly work on an alternative fetcher to know > whether this was a 0rtt request? Or is there another way? I seem to remember there was one but can't find it, so I may have been confused. With this said, it doesn't provide a big information since once the handshake is completed, it's exactly identical to a regular one. But it can be nice for statistics at least. Willy
Re: `ssl_fc_has_early` fetcher and 0rtt
Hello Willy, Thank you for your answer, On Wed, Sep 9, 2020 at 4:39 PM Willy Tarreau wrote: > If I remember well, the principle consists in detecting whether or not > the request was received using TLS early data (0-rtt) before the handshake > was completed. The problem is that early data may trivially be captured > and replayed, so you don't necessarily want to accept all of them, only > replay-safe requests. Typically a login page that is limited to 3 > attempts before blocking should not be allowed, but fetching a favicon > is totally safe. > > Once the handshake ends you'll know whether it was safe or not, so you > can actually decide to wait on this function to return false to indicate > that the request is complete and not replayed, or just use it to return > a "425 too early" response for certain sensitive resources. > > I think it's not easy to reproduce these tests, you need a high enough > latency between haproxy and the client so that the handshake is not > already completed when you evaluate the rule, and of course you need > to make sure the client sends using early data. I don't remember how > Olivier used to run his tests but I remember that it was a bit tricky, > so it's very possible that you never fall into the situation where you > can see the unvalidated early data yet. It means my understanding of this fetcher was wrong indeed. For me the protection was here: http-request wait-for-handshake if ! METH_GET and the fetcher here to log whether it was a 0rtt request or not. In reality, it means all our requests have completed the handshake when the rule is evaluated (which is surprising looking at the number we have). So maybe we can possibly work on an alternative fetcher to know whether this was a 0rtt request? Or is there another way? Thanks, -- William
Re: `ssl_fc_has_early` fetcher and 0rtt
Hi William! On Wed, Sep 09, 2020 at 12:02:03PM +0200, William Dauchy wrote: > On Wed, Sep 9, 2020 at 10:48 AM William Dauchy wrote: > > I'm trying to understand `ssl_fc_has_early` fetcher behavior as I'm > > unable to find a single request where it returns 1. > > (sorry, forgot to mention, all of these tests were done on v2.2.x) If I remember well, the principle consists in detecting whether or not the request was received using TLS early data (0-rtt) before the handshake was completed. The problem is that early data may trivially be captured and replayed, so you don't necessarily want to accept all of them, only replay-safe requests. Typically a login page that is limited to 3 attempts before blocking should not be allowed, but fetching a favicon is totally safe. Once the handshake ends you'll know whether it was safe or not, so you can actually decide to wait on this function to return false to indicate that the request is complete and not replayed, or just use it to return a "425 too early" response for certain sensitive resources. I think it's not easy to reproduce these tests, you need a high enough latency between haproxy and the client so that the handshake is not already completed when you evaluate the rule, and of course you need to make sure the client sends using early data. I don't remember how Olivier used to run his tests but I remember that it was a bit tricky, so it's very possible that you never fall into the situation where you can see the unvalidated early data yet. Hoping this helps, Willy
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 9, 2020 at 10:48 AM William Dauchy wrote: > I'm trying to understand `ssl_fc_has_early` fetcher behavior as I'm > unable to find a single request where it returns 1. (sorry, forgot to mention, all of these tests were done on v2.2.x) -- William
`ssl_fc_has_early` fetcher and 0rtt
Hello, I'm trying to understand `ssl_fc_has_early` fetcher behavior as I'm unable to find a single request where it returns 1. Our config has 0rtt enabled and it is as follow: ``` global log 127.0.0.1 format rfc5424 local0 info daemon stats socket /var/lib/haproxy/stats level admin mode 600 user haproxy group haproxy expose-fd listeners stats timeout 2m maxconn 524288 user haproxy group haproxy set-dumpable tune.bufsize 33792 tune.runqueue-depth 1200 tune.sched.low-latency on tune.ssl.cachesize 0 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES256-SHA ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384 ssl-server-verify none hard-stop-after 183s nbthread 16 cpu-map auto:1/1-16 0-15 spread-checks 5 chroot /etc/haproxy/chroot strict-limits defaults mode http log global option httplog option http-keep-alive option forwardfor except 127.0.0.0/8 option redispatch 1 option http-ignore-probes retries 3 retry-on conn-failure empty-response response-timeout 0rtt-rejected timeout http-request 10s timeout queue 1s timeout connect 10s timeout client 300s timeout server 300s timeout http-keep-alive 10s timeout check 5s maxconn 524288 balance roundrobin http-reuse always default-server inter 10s fastinter 1s fall 3 slowstart 20s observe layer7 error-limit 5 on-error fastinter pool-purge-delay 10s tfo allow-0rtt pool-low-conn 32 http-check expect rstatus ^(200|429) frontend fe_foo bind x:80 name http_ip process 1/all tfo bind x:443 name https_ip process 1/all ssl crt /etc/haproxy/tls/fe_foo alpn h2,http/1.1 tfo allow-0rtt log-format-sd [fc_rtt=\"%[fc_rtt]\"\ fc_0rtt=\"%[ssl_fc_has_early]\"\ fc_resumed=\"%[ssl_fc_is_resumed]\"] http-request disable-l7-retry if ! METH_GET http-request wait-for-handshake if ! METH_GET use_backend bar ``` - Am I right that tune.ssl.cachesize is not involved here? - What's odd is that I do find requests which returns 1 with `ssl_fc_is_resumed` fetcher. - I looked at `ssl_fc_has_early` and I did not found anything strange. - I tried to remove `wait-for-handshake` and `tune.ssl.cachesize 0` without success >From the client point of view I'm able to test it through: openssl s_client -tls1_3 -state -connect x:443 -servername foo.com -keylogfile keylogfile.log -sess_out ~/ssl/session.data openssl s_client -tls1_3 -state -connect x:443 -servername foo.com -keylogfile keylogfile.log -sess_in ~/ssl/session.data -early_data httpget.txt The second time I'm able to see: Reused, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 256 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was accepted Verify return code: 0 (ok) So to me, from the client point of view I'm able to make use of 0rtt. Any idea how to explain `ssl_fc_has_early` fetcher behaviour? Am I missing something in my config? Does someone have a different behaviour with a similar config? Best, -- William
