Re: Critique our use of curl; and HTTP/2, /3, multiplexing & pipelining questions

[Stefan Eissing via curl-library]

> Am 22.02.2023 um 17:01 schrieb Daniel Stenberg via curl-library 
> <curl-library@lists.haxx.se>:
> 
> On Tue, 21 Feb 2023, Richard W.M. Jones via curl-library wrote:
> 
>> Firstly, I don't understand if the multi interface would actually help us 
>> here.  Because nbdkit gives us lots of threads and expects an NBD request to 
>> be processed synchronously on that thread, using the easy interface is a 
>> natural .. easy(!) .. fit.
> 
> Doing multiple transfers using the multi interface instead of separate 
> threads using the easy interface has some pros and some cons.
> 
> Pros:
> 
> - shared pools and caches
> - you can multiplex transfers with h2/h3
> 
> Cons:
> 
> - multiple threads running on multiple cores can perform faster in case you
>   are CPU bound

There is no one size fits all here, as Daniel indicates. If you register 
callbacks in curl to get your answers and expect them to be called on a 
particular thread, you need to run curl in that thread.

The transfer sizes matter as well. If you have many small transfers, 
multiplexing can really help. For large transfers, the gains are less. But then 
again, if you, have a small request in the "middle" it will not need to wait on 
the large ones to complete.

As you can see, it really depends on your application, how it processes the 
responses, what kind of requests it makes and what response times it needs.

>> A second thing I'm unclear about with multi is whether the individual easy 
>> handles which are added are related in any way -- eg. if they all share the 
>> same TCP connection to the web server?
> 
> The *can* share the same conncetion with HTTP/2 and HTTP/3 multiplexing, yes.
> 
>> Reading the page makes me think this is not the case, the multi interface is 
>> just a way to group easy handles for the purposes of using a select/poll or 
>> event-driven API, and apart from that there is no relationship.
> 
> They also share connection pool, DNS cache, CA cert cache etc.
> 
>> The third and main concern is whether we are using curl most efficiently. In 
>> particular, whether we are using HTTP/2 (and in future HTTP/3) as 
>> efficiently as we could be (eg. exploiting multiplexing).
> 
> For multiplexing you *must* use the multi interface and add multiple 
> transfers to it.
> 
>> I notice that HTTP/1.1-style pipelining was removed from curl, and I suppose 
>> HTTP/2 multiplexing is meant to replace this.  However since we are using 
>> the easy interface and doing everything synchronously, it's my understanding 
>> that we are not exploiting multiplexing, unless curl itself does something 
>> clever internally.
> 
> It can't do anything clever when for synchronous serial transfers. Then you 
> can't do multiplexing.
> 
> -- 
> 
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
> -- 
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette:   https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html