Re: [TLS] Time to first byte vs time to last byte
On Sat, 9 Mar 2024 at 10:23, Bas Westerbaan wrote: > Given that especially for the web, CDNs used much higher initcwnds, > > > Please, let us not assume every website is behind a CDN. > Isn't that assumption reasonable? At least for global websites --- without CDN performance sucks. ___ TLS mailing list TLS@ietf.org https://www.ietf.org/mailman/listinfo/tls
Re: [TLS] Time to first byte vs time to last byte
> > Given that especially for the web, CDNs used much higher initcwnds, Please, let us not assume every website is behind a CDN. > let's focus on Figure 10. Based on Fig 10, 50-100KB of data over a PQ > connection, the TTLB would be 10-15% slower for 1Mbps and 200ms RTT. At > higher speeds, this percentage is much less (1-1.5% based on Fig 9b), but > let's focus on the slow link. > > If we consider the same case for handshake, then the PQ handshake slowdown > is 30-35% which definitely looks like a very impactful slowdown. A 10-15% > for the TTLB is much less, but someone could argue that even that is a > significant slowdown. Note we are still in a slow link, so even the > classical conn transferring 72KB is probably suffering. To quantify that I > looked at my data from these experiments. A classical connection TTLB for > 50-100KB of data at 1Mbps and 200ms RTT and 0% loss was ~1.25s. This is not > shown in the paper because I only included text about the 10% loss case. > 1.25s for a 72KB page to start getting rendered on a browser over a > classical conn vs 1.25*1.15=1.44s for a PQ one. I am not sure any user > waiting for 1.25s will close the browser at 1.44s. > > Btw, the Google PageSpeed Insights TTFB metric which includes (DNS lookup, > redirects and more) considers 0.8s - 1.8s as "Needs improvement". In our > experiments, the handshake time for 1Mbps and 200ms RTT amounted to 436ms > and 576ms for the classical and PQ handshakes respectively. I am not sure > the extra 140ms (30-35% slowdown) for the PQ handshake would even throw the > Google PageSpeed Insights TTFB metric to the "Needs improvement" category. > > > > -Original Message- > From: Martin Thomson > Sent: Thursday, March 7, 2024 10:26 PM > To: Kampanakis, Panos ; David Benjamin < > david...@chromium.org>; Deirdre Connolly ; Rob > Sayre > Cc: TLS@ietf.org; Childs-Klein, Will > Subject: RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > Hi Panos, > > I realize that TTLB might correlate well for some types of web content, > but it's important to recognize that lots of web content is badly bloated > (if you can tolerate the invective, this is a pretty good look at the > situation, with numbers: > https://infrequently.org/series/performance-inequality/). > > I don't want to call out your employer's properties in particular, but at > over 3M and with relatively few connections, handshakes really don't play > much into page load performance. That might be typical, but just being > typical doesn't mean that it's a case we should be optimizing for. > > The 72K page I linked above looks very different. There, your paper shows > a 20-25% hit on TTLB. TTFB is likely more affected due to the way > congestion controllers work and the fact that you never leave slow start. > > Cheers, > Martin > > On Fri, Mar 8, 2024, at 13:56, Kampanakis, Panos wrote: > > Thx Deirdre for bringing it up. > > > > David, > > > > ACK. I think the overall point of our paper is that application > > performance is more closely related to PQ TTLB than PQ TTFB/handshake. > > > > Snippet from the paper > > > > *> Google’s PageSpeed Insights [12] uses a set of metrics to measure > > the user experience and webpage performance. The First Contentful > > Paint (FCP), Largest Contentful Paint (LCP), First Input Delay (FID), > > Interaction to Next Paint (INP), Total Blocking Time (TBT), and > > Cumulative Layout Shift (CLS) metrics include this work’s TTLB along > > with other client-side, browser application-specific execution delays. > > The PageSpeed Insights TTFB metric measures the total time up to the > > point the first byte of data makes it to the client. So, PageSpeed > > Insights TTFB is like this work’s TTFB/TLS handshake time with > > additional network delays like DNS lookup, redirect, service worker > > startup, and request time.* > > > > Specifically about the Web, TTLB (as defined in the paper) is directly > > related to FCP, LCP, FID, INP, TBT, CLS, which are 6 of the 7 metrics > > in Google’s PageSpeed Insights. We don’t want to declare that TTLB is > > the ultimate metric, but intuitively, I think it is a better indicator > > when it comes to application performance than TTFB. > > > > That does not intend to underestimate the importance of the studies on > > handshake performance which was crucial to identify the best > > performing new KEMs and signatures. It a
Re: [TLS] Time to first byte vs time to last byte
Hi Martin, I think we are generally in agreement, but I want to push back on the argument that the PQ slowdown for a page transferring 72KB is going to be the problem. I will try to quantify this below (look for [72KBExample]). Btw, if you have any stats on Web content size distribution, I am interested. Other than averages, I could not find any data on how Web content size looks today. Note that our paper not bashing TTFB as a metric, we are just saying TTFB is more relevant for use-cases that send little data, which is not the case for most applications today. Snippet from the Conclusion of the paper > Connections that transfer <10-20KB of data will probably be more impacted by > the new data-heavy handshakes This study picked data sizes based on public data on Web sizes (HTTP Archive) and other data for other cloud uses. Of course, if we reached a world where most use-cases (Web connections, IoT sensor measurement conns, cloud conns) were typically sending <50KB, then the TTFB would become more relevant. I am not sure we are there or we will ever be. Even the page you referenced (thx, I did not know of it) argues " ~100KiB of HTML/CSS/fonts and ~300-350KiB of JS." from 2021. [72KBExample] I think your 20-25% for a 72KB example page probably came from reading Fig 4b which includes an extra RTT due to initcwnd=10. Given that especially for the web, CDNs used much higher initcwnds, let's focus on Figure 10. Based on Fig 10, 50-100KB of data over a PQ connection, the TTLB would be 10-15% slower for 1Mbps and 200ms RTT. At higher speeds, this percentage is much less (1-1.5% based on Fig 9b), but let's focus on the slow link. If we consider the same case for handshake, then the PQ handshake slowdown is 30-35% which definitely looks like a very impactful slowdown. A 10-15% for the TTLB is much less, but someone could argue that even that is a significant slowdown. Note we are still in a slow link, so even the classical conn transferring 72KB is probably suffering. To quantify that I looked at my data from these experiments. A classical connection TTLB for 50-100KB of data at 1Mbps and 200ms RTT and 0% loss was ~1.25s. This is not shown in the paper because I only included text about the 10% loss case. 1.25s for a 72KB page to start getting rendered on a browser over a classical conn vs 1.25*1.15=1.44s for a PQ one. I am not sure any user waiting for 1.25s will close the browser at 1.44s. Btw, the Google PageSpeed Insights TTFB metric which includes (DNS lookup, redirects and more) considers 0.8s - 1.8s as "Needs improvement". In our experiments, the handshake time for 1Mbps and 200ms RTT amounted to 436ms and 576ms for the classical and PQ handshakes respectively. I am not sure the extra 140ms (30-35% slowdown) for the PQ handshake would even throw the Google PageSpeed Insights TTFB metric to the "Needs improvement" category. -Original Message- From: Martin Thomson Sent: Thursday, March 7, 2024 10:26 PM To: Kampanakis, Panos ; David Benjamin ; Deirdre Connolly ; Rob Sayre Cc: TLS@ietf.org; Childs-Klein, Will Subject: RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi Panos, I realize that TTLB might correlate well for some types of web content, but it's important to recognize that lots of web content is badly bloated (if you can tolerate the invective, this is a pretty good look at the situation, with numbers: https://infrequently.org/series/performance-inequality/). I don't want to call out your employer's properties in particular, but at over 3M and with relatively few connections, handshakes really don't play much into page load performance. That might be typical, but just being typical doesn't mean that it's a case we should be optimizing for. The 72K page I linked above looks very different. There, your paper shows a 20-25% hit on TTLB. TTFB is likely more affected due to the way congestion controllers work and the fact that you never leave slow start. Cheers, Martin On Fri, Mar 8, 2024, at 13:56, Kampanakis, Panos wrote: > Thx Deirdre for bringing it up. > > David, > > ACK. I think the overall point of our paper is that application > performance is more closely related to PQ TTLB than PQ TTFB/handshake. > > Snippet from the paper > > *> Google’s PageSpeed Insights [12] uses a set of metrics to measure > the user experience and webpage performance. The First Contentful > Paint (FCP), Largest Contentful Paint (LCP), First Input Delay (FID), > Interaction to Next Paint (INP), Total Blocking Time (TBT), and > Cumulative Layout Shift (CLS) metrics include this work’s TTLB along > with other client-side, browser application-specific execution delays.
Re: [TLS] Time to first byte vs time to last byte
Hi Panos, I realize that TTLB might correlate well for some types of web content, but it's important to recognize that lots of web content is badly bloated (if you can tolerate the invective, this is a pretty good look at the situation, with numbers: https://infrequently.org/series/performance-inequality/). I don't want to call out your employer's properties in particular, but at over 3M and with relatively few connections, handshakes really don't play much into page load performance. That might be typical, but just being typical doesn't mean that it's a case we should be optimizing for. The 72K page I linked above looks very different. There, your paper shows a 20-25% hit on TTLB. TTFB is likely more affected due to the way congestion controllers work and the fact that you never leave slow start. Cheers, Martin On Fri, Mar 8, 2024, at 13:56, Kampanakis, Panos wrote: > Thx Deirdre for bringing it up. > > David, > > ACK. I think the overall point of our paper is that application > performance is more closely related to PQ TTLB than PQ TTFB/handshake. > > Snippet from the paper > > *> Google’s PageSpeed Insights [12] uses a set of metrics to measure > the user experience and webpage performance. The First Contentful Paint > (FCP), Largest Contentful Paint (LCP), First Input Delay (FID), > Interaction to Next Paint (INP), Total Blocking Time (TBT), and > Cumulative Layout Shift (CLS) metrics include this work’s TTLB along > with other client-side, browser application-specific execution delays. > The PageSpeed Insights TTFB metric measures the total time up to the > point the first byte of data makes it to the client. So, PageSpeed > Insights TTFB is like this work’s TTFB/TLS handshake time with > additional network delays like DNS lookup, redirect, service worker > startup, and request time.* > > Specifically about the Web, TTLB (as defined in the paper) is directly > related to FCP, LCP, FID, INP, TBT, CLS, which are 6 of the 7 metrics > in Google’s PageSpeed Insights. We don’t want to declare that TTLB is > the ultimate metric, but intuitively, I think it is a better indicator > when it comes to application performance than TTFB. > > That does not intend to underestimate the importance of the studies on > handshake performance which was crucial to identify the best performing > new KEMs and signatures. It also does not intend to underestimate the > importance of slimming down PQ TLS 1.3 handshakes as much as possible. > > Side note about Rob’s point: > We have not collected QUIC TTLB data yet, but I want to say that the > paper’s TTLB experimental results could more or less be extended to > QUIC be subtracting one RTT. OK, I don’t have experimental measurements > to prove it yet. So I will only make this claim and stop until I have > more data. > > > > *From:* TLS *On Behalf Of * David Benjamin > *Sent:* Thursday, March 7, 2024 3:41 PM > *To:* Deirdre Connolly > *Cc:* TLS@ietf.org > *Subject:* RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte > > *CAUTION*: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender > and know the content is safe. > > > This is good work, but we need to be wary of getting too excited about > TTLB, and then declaring performance solved. Ultimately, TTLB simply > dampens the impact of postquantum by mixing in the > (handshake-independent) time to do the bulk transfer. The question is > whether that reflects our goals. > > Ultimately, the thing that matters is overall application performance, > which can be complex to measure because you actually have to try that > application. Metrics like TTLB, TTFB, etc., are isolated to one > connection and thus easier to measure, and without checking each > application one by one. But they're only as valuable as they are > predictors of overall application performance. For TTLB, both the > magnitude and desirability of dampening effect are application-specific: > > If your goal is transferring a large file on the backend, such that you > really only care when the operation is complete, then yes, TTLB is a > good proxy for application system performance. You just care about > throughput in that case. Moreover, in such applications, if you are > transferring a lot of data, the dampening effect not only reflects > reality but is larger. > > However, interactive, user-facing applications are different. There, > TTLB is a poor proxy for application performance. For example, on the > web, performance is determined more by how long it takes to display a > meaningful webpage to the user. (We often call this the time to "first > contentful pai
Re: [TLS] Time to first byte vs time to last byte
Thx Deirdre for bringing it up. David, ACK. I think the overall point of our paper is that application performance is more closely related to PQ TTLB than PQ TTFB/handshake. Snippet from the paper > Google’s PageSpeed Insights [12] uses a set of metrics to measure the user > experience and webpage performance. The First Contentful Paint (FCP), Largest > Contentful Paint (LCP), First Input Delay (FID), Interaction to Next Paint > (INP), Total Blocking Time (TBT), and Cumulative Layout Shift (CLS) metrics > include this work’s TTLB along with other client-side, browser > application-specific execution delays. The PageSpeed Insights TTFB metric > measures the total time up to the point the first byte of data makes it to > the client. So, PageSpeed Insights TTFB is like this work’s TTFB/TLS > handshake time with additional network delays like DNS lookup, redirect, > service worker startup, and request time. Specifically about the Web, TTLB (as defined in the paper) is directly related to FCP, LCP, FID, INP, TBT, CLS, which are 6 of the 7 metrics in Google’s PageSpeed Insights. We don’t want to declare that TTLB is the ultimate metric, but intuitively, I think it is a better indicator when it comes to application performance than TTFB. That does not intend to underestimate the importance of the studies on handshake performance which was crucial to identify the best performing new KEMs and signatures. It also does not intend to underestimate the importance of slimming down PQ TLS 1.3 handshakes as much as possible. Side note about Rob’s point: We have not collected QUIC TTLB data yet, but I want to say that the paper’s TTLB experimental results could more or less be extended to QUIC be subtracting one RTT. OK, I don’t have experimental measurements to prove it yet. So I will only make this claim and stop until I have more data. From: TLS On Behalf Of David Benjamin Sent: Thursday, March 7, 2024 3:41 PM To: Deirdre Connolly Cc: TLS@ietf.org Subject: RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. This is good work, but we need to be wary of getting too excited about TTLB, and then declaring performance solved. Ultimately, TTLB simply dampens the impact of postquantum by mixing in the (handshake-independent) time to do the bulk transfer. The question is whether that reflects our goals. Ultimately, the thing that matters is overall application performance, which can be complex to measure because you actually have to try that application. Metrics like TTLB, TTFB, etc., are isolated to one connection and thus easier to measure, and without checking each application one by one. But they're only as valuable as they are predictors of overall application performance. For TTLB, both the magnitude and desirability of dampening effect are application-specific: If your goal is transferring a large file on the backend, such that you really only care when the operation is complete, then yes, TTLB is a good proxy for application system performance. You just care about throughput in that case. Moreover, in such applications, if you are transferring a lot of data, the dampening effect not only reflects reality but is larger. However, interactive, user-facing applications are different. There, TTLB is a poor proxy for application performance. For example, on the web, performance is determined more by how long it takes to display a meaningful webpage to the user. (We often call this the time to "first contentful paint".) Now, that is a very high-level metric that is impacted by all sorts of things, such as whether this is a repeat visit, page structure, etc. So it is hard to immediately translate that back down to TLS. But it is frequently much closer to the TTFB side of the spectrum than the TTLB side. And indeed, we have been seeing impacts from PQ to our high-level metrics on mobile. There's also a pretty natural intuition for this: since there is much more focus on latency than throughput, optimizing an interactive application often involves trying to reduce the amount of traffic on the critical path. The more the application does so, the less accurate TTLB's dampening effect is, and the closer we trend towards TTFB. (Of course, some optimizations in this space involve making fewer connections, etc. But the point here was to give a rough intuition.) On Thu, Mar 7, 2024 at 2:58 PM Deirdre Connolly mailto:durumcrustu...@gmail.com>> wrote: "At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web (MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a metric for assessing the total impact of data-heavy, quantum-resistant algorithms such as ML-KEM and ML-DSA on real-world TLS 1.3 connections. Our pa
Re: [TLS] Time to first byte vs time to last byte
This is good work, but we need to be wary of getting too excited about TTLB, and then declaring performance solved. Ultimately, TTLB simply dampens the impact of postquantum by mixing in the (handshake-independent) time to do the bulk transfer. The question is whether that reflects our goals. Ultimately, the thing that matters is overall application performance, which can be complex to measure because you actually have to try that application. Metrics like TTLB, TTFB, etc., are isolated to one connection and thus easier to measure, and without checking each application one by one. But they're only as valuable as they are predictors of overall application performance. For TTLB, both the magnitude and desirability of dampening effect are application-specific: If your goal is transferring a large file on the backend, such that you really only care when the operation is complete, then yes, TTLB is a good proxy for application system performance. You just care about throughput in that case. Moreover, in such applications, if you are transferring a lot of data, the dampening effect not only reflects reality but is larger. However, interactive, user-facing applications are different. There, TTLB is a poor proxy for application performance. For example, on the web, performance is determined more by how long it takes to display a meaningful webpage to the user. (We often call this the time to "first contentful paint".) Now, that is a very high-level metric that is impacted by all sorts of things, such as whether this is a repeat visit, page structure, etc. So it is hard to immediately translate that back down to TLS. But it is frequently much closer to the TTFB side of the spectrum than the TTLB side. And indeed, we have been seeing impacts from PQ to our high-level metrics on mobile. There's also a pretty natural intuition for this: since there is much more focus on latency than throughput, optimizing an interactive application often involves trying to reduce the amount of traffic on the critical path. The more the application does so, the less accurate TTLB's dampening effect is, and the closer we trend towards TTFB. (Of course, some optimizations in this space involve making fewer connections, etc. But the point here was to give a rough intuition.) On Thu, Mar 7, 2024 at 2:58 PM Deirdre Connolly wrote: > "At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web > (MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a > metric for assessing the total impact of data-heavy, quantum-resistant > algorithms such as ML-KEM and ML-DSA on real-world TLS 1.3 connections. Our > paper shows that the new algorithms will have a much lower net effect on > connections that transfer sizable amounts of data than they do on the TLS > 1.3 handshake itself." > > > https://www.amazon.science/blog/delays-from-post-quantum-cryptography-may-not-be-so-bad > > ¹ > https://www.amazon.science/publications/the-impact-of-data-heavy-post-quantum-tls-1-3-on-the-time-to-last-byte-of-real-world-connections/ > > > ___ > TLS mailing list > TLS@ietf.org > https://www.ietf.org/mailman/listinfo/tls > ___ TLS mailing list TLS@ietf.org https://www.ietf.org/mailman/listinfo/tls
Re: [TLS] Time to first byte vs time to last byte
I agree the efficiency concerns are generally overstated, but this study should have measured QUIC etc, since the web pages will have all sorts of awful performance problems. But the thing you have to watch out for is when someone in the datacenter steps on the power cord or something (or DNS is wrong, etc). Then, you get a stampede of clients reconnecting, and there you really do care about how expensive the handshake is. thanks, Rob On Thu, Mar 7, 2024 at 11:58 AM Deirdre Connolly wrote: > "At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web > (MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a > metric for assessing the total impact of data-heavy, quantum-resistant > algorithms such as ML-KEM and ML-DSA on real-world TLS 1.3 connections. Our > paper shows that the new algorithms will have a much lower net effect on > connections that transfer sizable amounts of data than they do on the TLS > 1.3 handshake itself." > > > https://www.amazon.science/blog/delays-from-post-quantum-cryptography-may-not-be-so-bad > > ¹ > https://www.amazon.science/publications/the-impact-of-data-heavy-post-quantum-tls-1-3-on-the-time-to-last-byte-of-real-world-connections/ > > > ___ > TLS mailing list > TLS@ietf.org > https://www.ietf.org/mailman/listinfo/tls > ___ TLS mailing list TLS@ietf.org https://www.ietf.org/mailman/listinfo/tls
