RE: Curl_poll and Windows
Sorry if the formatting is messed up. Microsoft Outlook doesn't handle
text-only mailing lists very well.
> From: Daniel Stenberg
> Sent: Monday, November 18, 2019 2:11 PM
>
> > First, Windows Vista and later have an API called WSAPoll. It works
> > almost the same as UNIX poll.
>
> Almost is the key there. We once used it and ditched it due to the "almost"
> not being good enough. I blogged about that back in the day
> https://urldefense.com/v3/__https://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/__;!2E0gRdhhnqPNNL0!3hnobk8HPrx1rr6DPng7kJRSEOkCMRzCyqWBTYHnF9Ujdkn0MiNtF_t6DnFX1ha9$
>
>
> Not saying it can't be used, it just needs to be done with care and attention
> to details.
Yikes. Thank you for pointing that out. It saved me from making a big mistake
in my code >.<
It's ridiculous that this has been broken since 2006 (Vista) without this
behavior being documented. I was rather mad at Microsoft when I read your
email yesterday >.<
> > FD_SET is really slow in Windows.
>
> So slow it actually matters and is worth it for us to replace it with
> something else you think?
In Windows, FD_SET takes O(n) time where n is the number of sockets already in
the fd_set. So in a loop to add a bunch of sockets to an fd_set, it's O(n^2).
But that only really matters if there are a lot of curl connections at once.
> We'll happily welcome patches that improve libcurl, in this area as well as
> others! This is all news to me and yet I've toyed with these things for a
> while... albeit, not much on Windows.
Sure, I'll consider it. The only issue that comes to mind is, in curl-land,
how would I malloc() memory that is aligned to at least _Alignof(SOCKET)? I
suppose that this ought to happen automatically if the hosting application's
malloc is sane.
> > A correct Windows implementation of Curl_wait_ms that is intended to
> > mimic
> > select() would use SleepEx with the second parameter TRUE, in a loop
> > because unlike select(), SleepEx will abort if an APC occurs.
>
> I don't even know what an APC is, but again, we welcome pull requests! =)
>
> I'm not aware of our use of Sleep() there have ever been an issue. That could
> of course also be because it doesn't matter much if the sleep would return
> prematurely once every blue moon.
An APC ("asynchronous procedure call") is a lot like a signal with a signal
handler. When some operation completes, the APC will trigger on the thread
that requested it. The big difference from UNIX signals is that APCs don't get
dispatched unless the target thread is ready for them: the thread has to be in
an "alertable wait state". The thread must be executing one of the many
waiting functions with a flag saying that APCs are OK during this wait.
Windows's Sleep(n) is the same as SleepEx(n, FALSE). Sleep therefore won't
allow APCs to be dispatched during the wait. This is different from select(),
which is always an alertable wait state. The difference is rather minor, and
nothing in curl uses APCs. I doubt it's caused problems so far that Sleep(n)
isn't equivalent to waiting n milliseconds in select().
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: How to manage buggy http/2 in MacOS and Linux
On 11/19/2019 8:00 AM, Jeroen Ooms wrote: On Tue, Nov 19, 2019 at 9:25 AM Ray Satiro via curl-library wrote: On 11/18/2019 8:42 AM, Jeroen Ooms via curl-library wrote: I maintain the libcurl R bindings which are used by 1M+ users to build clients for countless web services. On MacOS and Linux, the bindings link to the system version of libcurl. As of MacOS Catalina (released last month) this is now libcurl 7.64.1 which is the first time that http/2 is enabled by default. This is where the problems begin. Users that have upgraded to Catalina are reporting applications that were stable before are now randomly giving "Error in the HTTP2 framing layer" errors. Similar sounds from Linux users. As we know, HTTP/2 support in libcurl was quite buggy until very recently (eg: https://daniel.haxx.se/blog/2018/09/05/curl-7-61-1-comes-with-only-bug-fixes/ ) and maybe still today not as reliable as HTTP/1. As maintainer of the bindings, I'm not sure how to handle this. One solution would to be override the default CURLOPT_HTTP_VERSION in the bindings to CURL_HTTP_VERSION_1_1 for certain versions of libcurl, but it's hard to judge which versions of libcurl have robust http/2 support. "Quite buggy" is an overstatement. Sorry, I didn't mean to be overly critical, again http/2 is incredibly complex. But in my experience, http/2 in libcurl has only very recently become more stable. In Daniel's words from the blog 1 year ago: "I think it is safe to say that HTTP/2 users of libcurl have previously used it in a pretty “tidy” fashion, because I believe I corrected four or five separate issues that made it misbehave. It was rather pure luck that has made it still work as well as it has for past users!". Obviously, most problems only appear when you start multiplexing, they do not really affect single requests over the command line. I think Daniel calling it "pure luck" is too conciliatory, it was that edge cases are just that, edge cases. I am subscribed to your repo and I am not aware of issues regarding this. Without knowing what was reported I think it's premature to say. Most of these problems are not reported in my repo, I get emailed or tagged by users who attribute the error to the server or the package that uses curl. But if you search for "Error in the HTTP2 framing layer" there are a lot of people starting to see these errors recently, without realizing these problems originate. Here is one example of such a report I was trying to debug: https://github.com/eddelbuettel/rpushbullet/issues/57 I don't have statistics on this but I'll guess most of those "framing layer" errors are proper. Some servers don't hangup properly and that is a problem for HTTP/2. Nevermind that TLS on its own has similar sensitivity. The blame certainly can be on curl, I think it's few and far between. For example #4267 which was fixed [1] in 7.67.0. That is a patch that you would want to backport if you were an OS maintainer maintaining a recent older version of curl. [1]: https://github.com/curl/curl/commit/c1b6a38 --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Curl as static library with dependencies
We use MinGW on windows. We need a static build of curl with all references to -OpenSSL 1.1.1d -brotli 1.0.7 -libssh2 1.9.0 -nghttp2 1.39.2 -zlib 1.2.11 We followed your doc, but we weren't able to do it. Can you provide us static curl build with all those dependencies? Many thanks, Andrea --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: How to manage buggy http/2 in MacOS and Linux
On Tue, Nov 19, 2019 at 9:25 AM Ray Satiro via curl-library wrote: > > On 11/18/2019 8:42 AM, Jeroen Ooms via curl-library wrote: > > I maintain the libcurl R bindings which are used by 1M+ users to build > > clients for countless web services. On MacOS and Linux, the bindings > > link to the system version of libcurl. As of MacOS Catalina (released > > last month) this is now libcurl 7.64.1 which is the first time that > > http/2 is enabled by default. This is where the problems begin. > > > > Users that have upgraded to Catalina are reporting applications that > > were stable before are now randomly giving "Error in the HTTP2 framing > > layer" errors. Similar sounds from Linux users. As we know, HTTP/2 > > support in libcurl was quite buggy until very recently (eg: > > https://daniel.haxx.se/blog/2018/09/05/curl-7-61-1-comes-with-only-bug-fixes/ > > ) and maybe still today not as reliable as HTTP/1. > > > > As maintainer of the bindings, I'm not sure how to handle this. One > > solution would to be override the default CURLOPT_HTTP_VERSION in the > > bindings to CURL_HTTP_VERSION_1_1 for certain versions of libcurl, but > > it's hard to judge which versions of libcurl have robust http/2 > > support. > > > "Quite buggy" is an overstatement. Sorry, I didn't mean to be overly critical, again http/2 is incredibly complex. But in my experience, http/2 in libcurl has only very recently become more stable. In Daniel's words from the blog 1 year ago: "I think it is safe to say that HTTP/2 users of libcurl have previously used it in a pretty “tidy” fashion, because I believe I corrected four or five separate issues that made it misbehave. It was rather pure luck that has made it still work as well as it has for past users!". Obviously, most problems only appear when you start multiplexing, they do not really affect single requests over the command line. > I am subscribed to your repo and I am not aware of issues regarding this. > Without knowing what was reported I think it's premature to say. Most of these problems are not reported in my repo, I get emailed or tagged by users who attribute the error to the server or the package that uses curl. But if you search for "Error in the HTTP2 framing layer" there are a lot of people starting to see these errors recently, without realizing these problems originate. Here is one example of such a report I was trying to debug: https://github.com/eddelbuettel/rpushbullet/issues/57 --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 problem: -7 Unable to send FXP_CLOSE command
On Tue, 19 Nov 2019, Christian Schmitz via curl-library wrote: Has anyone seen this error with SFTP file upload? I haven't. But it looks like a failure to send the SFTP command within libssh2 and it looks rather serious (ie not really any way to recover): https://github.com/libssh2/libssh2/blob/96bbefb2ded94ff3f8949ef298150a32fca40cc4/src/sftp.c#L2610-L2614 -- / daniel.haxx.se | Get the best commercial curl support there is - from me | Private help, bug fixes, support, ports, new features | https://www.wolfssl.com/contact/ --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: How to manage buggy http/2 in MacOS and Linux
On 11/18/2019 8:42 AM, Jeroen Ooms via curl-library wrote: I maintain the libcurl R bindings which are used by 1M+ users to build clients for countless web services. On MacOS and Linux, the bindings link to the system version of libcurl. As of MacOS Catalina (released last month) this is now libcurl 7.64.1 which is the first time that http/2 is enabled by default. This is where the problems begin. Users that have upgraded to Catalina are reporting applications that were stable before are now randomly giving "Error in the HTTP2 framing layer" errors. Similar sounds from Linux users. As we know, HTTP/2 support in libcurl was quite buggy until very recently (eg: https://daniel.haxx.se/blog/2018/09/05/curl-7-61-1-comes-with-only-bug-fixes/ ) and maybe still today not as reliable as HTTP/1. As maintainer of the bindings, I'm not sure how to handle this. One solution would to be override the default CURLOPT_HTTP_VERSION in the bindings to CURL_HTTP_VERSION_1_1 for certain versions of libcurl, but it's hard to judge which versions of libcurl have robust http/2 support. "Quite buggy" is an overstatement. I am subscribed to your repo and I am not aware of issues regarding this. Without knowing what was reported I think it's premature to say. If you are aware of a specific version you need to work around then you can use version info's version_num and nghttp2_ver_num, however the latter is only available since 7.66.0. Both have format (MAJOR << 16) | (MINOR << 8) | PATCH. curl_version_info_data *ver = curl_version_info(CURLVERSION_NOW); ver->version_num < 0x074300 /* libcurl 7.67.0 */ || ver->age < 5 /* libcurl 7.66.0 */ || ver->nghttp2_ver_num < 0x012600 /* nghttp2 1.38.0 */ then fall back to http 1.1 (age check isn't necessary above since the checked curl version is greater than that, but left in for example) I suppose it's also possible to report the problem you are having to Apple, they may fix it if they backport patches like other OS maintainers do. --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
