On 30/06/2018 5:16 AM, Daniel Stenberg wrote: > We got a fresh new HTTP pipelining bug submitted today [1]. We already > have other pipelining bugs listed in the KNOWN_BUGS document [2].
I am the submitter of that bug, and now that I understand it better, I have revised the issue report. It is NOT a pipelining bug, but a plumbing bug. Internally libcurl uses both a splay tree and a pending list to consider which request should be started when one ends. When new jobs get added to the splay tree with EXPIRE_NOW, they will be considered before already pending jobs on the pending list, which means they are executed out-of-order. The updated issue report has more detail about the circumstances of this bug. (https://github.com/curl/curl/issues/2701#issuecomment-408717419) > HTTP Pipelining is badly supported by curl in the sense that we have > bugs and it is a fragile feature without enough tests. Having looked at the code and having considered the previous HTTP/1.1 pipelining work documented on the mailing list (Carlo Wood and Monty Brandernberg in 2014, Linus Nielsen Feltzing and Joe Mason in 2013, Dmitry Kurochkin in 2008, likely others I overlooked, apologies), it looks like HTTP/1.1 pipelining is very well supported and tested, especially relative to other "features" that have crept into libcurl, that are far less well used and abused, like ssh, smtp, gopher, etc. > How many users do we have that use Pipelining? > > In the annual curl survey we ask users and a shockingly large amount of > users claim they use pipelining. Over 20% in the latest poll. I've never > trusted this number though, since we would have way more bug reports and > work done on the code if it truly was used that much. Also scanning through the mailing list and the issue tracker, I found there are several users of HTTP pipelining, and the sophistication of their use-cases exceed that of the cut-and-paste users of libcurl. It looks to me like 20% is a very plausible number: * Sunil Sayyaparaju (Nov 9, 2017) wants to enable pipelining for idempotent POST requests, and modified his own copy (forked) just like [email protected] did 5 years prior, see below. * Evgen Bodunov (Nov 21, 2017) using pipelining to download map tiles with a iOS app (probably a mobile app), wants to cancel already issued requests. * Alexandre Bouin (May 3, 2016) is using pipelining to maximize throughput, has a good understanding of bandwidth, latency and network utilization concepts. * Cristian Morales Vega (Jun 18, 2015) using 7.43.0 with pipelining and the curl_multi_socket_action API, ran into the same out-of-order problem that I just reported at #2701, when using 7.58.0, and submitted a test case. Also had a similar conversation with badger about the order-of-execution guarantee that I had in issue #2701. Even the cause was narrowed down to the splay tree, just as I did. * Xavi Artigas (Mar 31, 2014) Using libcurl pipelining in a downloader library, to maximize throughput. Encountered corrupted downloads in libcurl 7.32 and 7.36 * Török Edwin (Jan 24, 2013), described a bug in 7.28.1 about pipelining and CONNECTTIMEOUT, and submitted a test case. * Yamin Zhou (Feb 19, 2014), wanted to use pipelining with libcurl-7.22.0, found it buggy, then upgraded to 7.35.0, and his app worked as expected. * David Strauss (Apr 10, 2013), wants to implement AAAA records and balancing/fail-over multiple hosts in combination with connection management and pipelining. Offered to sponsor the the work. The application is a FuseDav filesystem client. * Glen Milner (Jul 18, 2012), implementing pipelining and digest authentication, found a bug, looks like it was fixed by Joe Mason. * ru-17 <[email protected]> (Feb 27, 2012), modified his own libcurl source to enable pipelining for POST * Mario Castelan Caestro (July 14, 2011), question about the callback order. * Anton Serdyuk and Marcin Adamski (Oct 7, 2011), found an infinite loop in Curl_removeHandleFromPipeline (was already fixed in the repo HEAD at the time). The application is a web crawler. * Stefan Krause (Nov 16, 2010), had questions about pipelining and local ports. * Justin Skirts (July 20, 2009), wanted to implement flow control in a pipelined fetch. It's worth noting that these people generated a bit of mailing list activity at the time they were working on their respective projects, but are not regular contributors. I think this is typical of a developer to join the mailing list only when he has some kind of issue, but not stick around after he finds the solution. After all, libcurl is not their main interest, but just another library to get their project moving. In my experience, pipelining works very well in libcurl, aside from some configuration constraints. I expect that there are many more users of pipelining for whom a basic pipelining works well enough that they didn't need to ask questions on the mailing list. Many of those can likely tell if their application is indeed getting the expected performance from pipelining (Ie, I assume that other developers are reasonably intelligent) Also I note that none of these people appear to use libcurl in browser projects, but rather in one of several project types: - web crawler - some other backend communications (event logging like) - as transport for another application (DAV) - one instance of video streaming app (can't find the reference at the moment) - process control and automation (no examples on the ML, I have encountered many examples in my professional life), ie, embedded HTTP servers, used to control industrial equipment, test and measurement instrumentation, etc. It looks like the users who are interested in HTTP 1.1 pipelining are more keen on performance and invariably quote throughput (**not latency**) as the the reason for their need to pipeline. Furthermore, some of the applications I mentioned require a specified order of request execution, which is explicitly not available in HTTP/2. In order to use HTTP/2, these applications would require workarounds to issue only one request at a time in order to guarantee ordering, thus defeating the intent of HTTP/2, and achieving a lower performance than they could with HTTP 1.1 and pipelining. Such apps have no reason to consider HTTP/2. HTTP 1.1 will, in my opinion, continue to live along side HTTP/2 for the forseable future. While HTTP/2 has some benefits in browsing applications, it is its lack of pipelining that make is unsuitable for a host of other applications, like process control (request ordering required), games (many non-idempotent requests), or streaming (presentation-layer buffering issues). It may be a replacement for HTTP 1.1 in browsers, but an undesireable protocol in other applications. Multiplexing, which HTTP/2 offers, is **not pipelining**. Multiplexing is a way of performing tasks in parallel, pipelining is a way of performing tasks serially. Game client often rely on communicating game state with the game server, and the messages are often non-idempotent requests, and they need to be delivered to the server in a specific order. Pipelining guarantees this order. HTTP/2 by design and desire does not guarantee order of execution: "Note: Stream dependencies and weights express a transport preference, not a requirement, and as such do not guarantee a particular processingor transmission order. That is, the client cannot force the server to process the stream in a particular order using stream prioritization. While this may seem counterintuitive, it is in fact the desired behavior. We do not want to block the server from making progress on a lower priority resource if a higher priority resource is blocked." [1] Media Streaming applications, especially network-adaptive ones, typically need chunks of content delivered in the correct order. Delivering multiple chunks simultaneously, like HTTP/2 does, is the opposite of what such apps need. More than this, it is inefficient for any part of a later chunk to be delivered before the earlier chunk is completely delivered, as it would slow down the delivery of the earlier chunk, and require additional memory to buffer the later chunk which arrives early. It is precisely the pipelining feature of HTTP 1.1 which helps streaming apps both maximize throughput and minimize local buffering requirements. Chunks arrive in the order requested with pipelining. Take an app like netflix. It starts streaming a low-resolution chunk, so the presentation can start as soon as possible. While the first(s) chunks are delivered, the app can estimate the available network bandwidth, so the 2nd chunk requested can be of appropriate resolution to match network capabilities. But still, none of the 2nd chunk is needed until the 1st is completely delivered. This class of app heavily prefers HTTP 1.1 pipelining over HTTP/2 magic. While it can still work with HTTP/2, it requires more app development and more buffering space to deliver a smooth stream over HTTP/2. Or, if the available buffering space is limited, such as in embedded devices, buffering capacity wasted due to HTTP/2 delivering chunks simultaneously/overlapping means these embedded devices would be forced to display a lower resolution than they could with better buffer control. Case in point, Viren Baraiya, on Nov 14, 2017, while Engineering Manager at Netflix responded in a github issue [2] that Netflix were not working on HTTP/2 support. Pipelining is here to stay; in the HTTP 1.1 specification, in server implementations, and in other applications (perhaps) other than browsers. Whether libcurl continues to support it or not will only affect whether libcurl remains relevant for the non-browser, HTTP world. Remember HTTP is not HTML. The HTTP protocol is used for purposes much more varied than to deliver HTML and the associated stylesheets and images. I think what is likely to happen is that someone will fork libcurl, bring it back to its core functionality, bring it into RFC compliance, and clean up the cruft, and similar to what happened to OpenSSL/LibreSSL, while libcurl is free to continue addressing the needs of a browser-developer audience, even though it is not used in any browsers. Certainly if libcurl still supports a long defunct protocol like gopher, it should still support pipelining which was only revised 4 years ago (HTTP 1.1 RFCs were published June 6, 2014). In any case, what happens with HTTP 1.1 should not depend on the fate of HTTP/2. They are complementary, not evolutionary protocols. [1] https://github.com/Netflix/conductor/issues/363#issuecomment-344415181 [2] https://developers.google.com/web/fundamentals/performance/http2/ ------------------------------------------------------------------- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
