On 06/08/2018 4:53 PM, Daniel Stenberg wrote: > Right, quite ironic I think that it wasn't a pipelining bug that finally triggered me to put pipelining up for deprecation...
You keep saying there are lots of bugs and missing test cases. Since I am interested in pipelining, I did some research on this ML and through the issue tracker, and found no evidence of remaining bugs in the pipelining functionality. I did find evidence that lots of other developers are interested in using pipelining in their projects. This was the point of me writing the (long) summary. The state of the feature looks quite solid to me. Having looked at the inner workings of multi.c, it's a miracle that it works, but it mostly does work. > Several of those persons you mentioned idenfied problems or flaws with pipelining, several of those were never fixed. As we don't get them reported more I take as an indication that not enough users are using the feature to hit them that often (to report them). I have looked at every one of those reports, and I found no unresolved bugs. Care to list the remaining unaddressed issues? I noted that some are looking for functionality which does not exist in the protocol (like cancelling already issued requests). It's not clear to me what those devs expect would happen in that scenario; Also I don't know why they expect this functionality to exist. In most of computer science and internet protocols, there isn't much "unshoot the arrow" type of functionality; why do they expect it in curl? This being said, I do know there are some segfaults and crashes when attempting "unexpected" API calls, like removing active easy handles, or cleaning up the multi handle at "the wrong time". These are real bugs, but they are not related to pipelining. They are due to the mess of data members sprawled across unrelated internal structures. IMO, patching individual segfaults is futile, the messy data organization would just cause new bugs to pop up in other places, either immediately or at a future time. The test suite is not thorough enough to catch regressions, and I'm not talking about pipelining, but generally. > I know several "browser projects" as well that use libcurl. Or at least used to - I'm pretty bad at keeping track of how libcurl-using apps change over time... They are a minority. The main attraction of libcurl is in unattended, automated work involving the HTTP protocol, not in supporting an HTML rendering engine. Think yum, apt-get, git, etc. >> It looks like the users who are interested in HTTP 1.1 pipelining are more keen on performance and invariably quote throughput (**not latency**) as the the reason for their need to pipeline. > > Pipelining does *nothing* for throughput. It only (marginally) helps latency. I cannot agree with you. Pipelining enables full-duplex HTTP comm. Without it, the two parties would have a half-duplex channel, each waiting for the other to be done talking before talking itself. Surely you understand that each direction will experience higher throughput if the wait is eliminated. This was the drive behind HTTP/1.1 pipelining, not latency. Latency was never a problem, a browser could open as many TCP connections as needed to request objects in parallel and minimize latency, and still can, even if each connection is pipelined. In fact pipelining increases latency, as new requests are queued after those already in the pipeline. A queue increases latency, by definition, as each request is not sent immediately, but after waiting in queue. Connection persistence lowers latency in the case of serialized/ordered requests by doing away with connection setup. Pipelining increases latency, connection persistence lowers it. When combined, one can calculate what happens to latency. It depends on the size of each request, the pipeline depth, bandwidth, etc. > IMHO, HTTP/2's multiplexing is superior to pipelining in every way, and it has none of the downsides HTTP/1.1 pipelining has. > >> "Note: Stream dependencies and weights express a transport >> preference, not a requirement, and as such do not guarantee a >> particular processingor transmission order. > > I see where you found this quote, and while there's no way to absolutely force the order, RFC 7540 says: > > Inside the dependency tree, a dependent stream SHOULD only be allocated > resources if either all of the streams that it depends on (the chain of > parent streams up to 0x0) are closed or it is not possible to make progress > on them. > The use of SHOULD there does of course not prevent servers from disobeying it. But it would be against the advice of the spec. So strict ordering is not guaranteed, like it is in pipelining? For applications that *require* strict ordering and full link utilization, there is no substitute for HTTP/1.1 pipelining. HTTP/2 needs a wrapper to defeat all the fancy multiplexed "stream" logic in order that strict ordering be guaranteed. >> I think what is likely to happen is that someone will fork libcurl, bring it back to its core functionality, bring it into RFC compliance, and clean up the cruft > > Aha! So that mysterious someone is just waiting for me to cut off pipelining support *first*, then that excellent person will step forward and get working on fixing it? That work we've been hoping would happen for many years by now? > > It makes me wonder why this good resource doesn't step up already now so that they can save themselves quite a lot of trouble. A lot of devs venture on your mailing list, and are turned away, like I am. I offered to rewrite multi.c and contribute it back, but there is no interest from you apparently. If I decide to keep using libcurl (I don't have feasible alternatives), I will be maintaining my own private fork with multi.c cleaned up. I have noted in my summary that others appear to have taken the same route (wrt. pipelining POST requests). There are forks of libcurl being maintained and used, just nobody has yet volunteered to maintain their own fork as a lightweight, packageable alternative to libcurl yet. I don't think I have this calling either. >> while libcurl is free to continue addressing the needs of a browser-developer audience, > > Is that really what you think we're doing? Or are you just trying to be rude? And lack of pipelining support is what going to make that sudden big change? > I have observed that every time pipelining is mentioned, you promote HTTP/2 and pile on the FUD about HTTP/1.1, saying the former is 'safer' than the later. You cite the major browser's disabling pipelining and adopting HTTP/2. URL parsing is not apparently RFC compliant, but WHATWG compliant (ie, a browser interest group). Your libcurl release schedule is fast and furious, like the browsers. Judging from the seriousness of bugs in the github issue tracker, the releases contain alpha-level code. For a foundation library, this is unacceptable to many projects. Foundational = many important packages depend on it, like the package managers of debian and offshoots, redhat, etc, git, I'm sorry if I seem rude to you, but the arguments you make are (to me) clearly from a narrow, browser-specific perspective. My main professional experience is at the opposite end of tech, in control/automation/high-performance computing, where things remain stable for a long time. Your arguments are very fallacious from my perspective, and I don't buy them for a second. You are entitled to your perspective. Libcurl is your project, so you can dismiss my perspective. In April 2016, you proposed removing pipeling support, and said "Can we really remove it? Oh heck, we can do whatever we want.". You're free to ignore evidence that many projects do in fact rely on pipelining as a source of performance, and substitute your own fabricated FUD. It certainly doesn't appear that you're looking for feedback (in fact you dismiss your own survey results), but for support of your decisions. Just do whatever you want, why waste time asking the ML, if you don't want the feedback? > So you've written this huge email to basically say I'm wrong and pipelining is great but not a single word on what you can bring to the table to improve its weaknesses or how you can help out to make it better so that it we can keep it around with pride. In bug 2701 which you closed, I offered to contribute back my rewritten multi.c, that would guarantee order of execution. I provided a test case, but I don't have time at the moment to work on multi.c. You closed it. In bug 2782 I pointed out the examples with external event loops contain a use-after-free/segfaulting bug (which took me at least 1 day of hair pulling to debug), and provided a patch. You rejected it. Sorry to say, but you reject offered help and spew FUD. That is my impression. As far as I can see, the current implementation of pipelining in libcurl does not need bugfixes other than the out-of-order execution bug, and it is the bug I intended to fix and contribute back, until you closed issue 2701. > I'm prepared to reconsider or reschedule the deprecation plans for pipelining, but that will require that someone actually shows - with working code - that they're ready to help out. We need more test cases (especially such ones that break the pipes) and we need bug fixes to fix the known flaws (like removing a handle that is used in a pipe) and the ones the new test cases will find. Ideally some code could even be improved so that it can co-exist easier with non-pipelining code. > > If nobody steps up to this, I will take that as yet another indication that not that many people actually use or care about Pipelining support in libcurl. Removing an active handle from a pipeline is hardly a pipelining bug. I think you should maintain a list of pipelining bugs that need fixing, and test case scenarios that you know to break pipes. Who better to know what they are? I have listed many names and dates when various people have expressed a need for pipelining. Do you expect them to keep reading the ML and periodically reassure you that they are still using it? Regards. ------------------------------------------------------------------- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html