> Why does it have to be a new option? Why not just make the existing one > possible to set larger than today?
The problem here is that for H1/H2 buffering and flow control are controlled by TCP receive buffer size socket option, which tells kernel how to operate TCP stack for that socket. Whereas for H3 QUIC, it is controlled by client defined read buffer size, and not by the kernel. So, increasing curl handle read buffer size, for example, to 1MB helps H3, but has no effect on TCP flow control in H1/H2 unless socket options are also changed to set receive buffer to 1MB. And if, hypothetically, we wanted to provide one option for all protocols, then for H1/H2 we would need to set TCP receive buffer size socket option to match curl handle's read buffer size under the hood. There is also another subtle moment. In H1/H2, Curl's internal buffering is separated from transport layer buffering and flow control (which is managed by TCP stack), so curl handle read buffer size mostly affects how frequently the "write callback" is called (and maximal size of the passed data) without affecting TCP transport layer. This makes it possible in H1/H2 to have frequent "write callback" calls with small data blocks, while providing large buffering on TCP level. In H3, Curl's internal buffering also controls QUIC internal buffering, so increasing curl handle read buffer size besides affecting QUIC performance, will also affect the frequency of "write callback" calls and the sizes of data blocks passed to the "write callback". In other words, to get good download performance with H3, client will have to increase curl handle read buffer size and be ready to handle much larger data blocks in less frequent "write callback" calls compared to H2/H3. This may force client apps to have different handlings of H3 and H1/H2 caused by changed "write callback" frequency and increased data size. The existing separation between Curl handle's internal buffering and TCP transport layer buffering in H1/H2 made me think about a special "QUIC" option controlling QUIC buffering and flow control, while keeping the "read buffer size" option separate from that - so as in H1/H2 it will only control the frequency and size of data blocks used in the "write callback" calls. And while I think that your change is very helpful to have big download buffers in H3, we may still probably need to think about if we want to use CURLOPT_BUFFERSIZE option to control H3/QUIC transport buffering, considering the side effects I mentioned above. Thanks, Dmitry Karpov -----Original Message----- From: curl-library <curl-library-boun...@lists.haxx.se> On Behalf Of Daniel Stenberg via curl-library Sent: Saturday, January 7, 2023 7:07 AM To: Daniel Stenberg via curl-library <curl-library@lists.haxx.se> Cc: Daniel Stenberg <dan...@haxx.se> Subject: [EXTERNAL] Re: H3/QUIC flow control/buffering problem and suggestion On Sat, 7 Jan 2023, Daniel Stenberg via curl-library wrote: > If an application is willing to use N megabytes for a transfer buffer > when doing h3, surely they can then also be allowed to set the same > size for > non-h3 transfers? Proposal: https://github.com/curl/curl/pull/10256 -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
