Re: libcurl - MacOS 10.14 ...
On Tue, Nov 12, 2019 at 9:37 PM Jeroen Ooms wrote: > > On Tue, Nov 12, 2019 at 8:51 PM Daniel Stenberg via curl-library > wrote: > > > > On Tue, 12 Nov 2019, Bryan O'Brien via curl-library wrote: > > > > > Courtesy of curl_version(), heres the version I’m using > > > > > > libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1 > > > > > > If anyone is aware of a more recent version, or can point me towards some > > > newbie build instructions, I’d appreciate it. > > > > You build and install libcurl the same way on mac as on any *nix: > > > > $ ./configure > > $ make > > $ make test (optional) > > $ make install > > > > For more details, see https://curl.haxx.se/docs/install.html > > > > There are also package managers for mac as well, like homebrew, but it's not > > clear to be if they offer libcurl installs (like with headers etc). > > Yes they do. There are two versions: one build with macos secure > transport, and one build with openssl+nghttp2: > > - https://github.com/homebrew/homebrew-core/blob/master/Formula/curl.rb > - > https://github.com/homebrew/homebrew-core/blob/master/Formula/curl-openssl.rb > > Because this homebrew package would conflict with a native macos > version of curl, it is not automatically linked into /usr/local, so > you need to set PKG_CONFIG_PATH. For example use this to build the > crawler example against the latest curl with http2 support: Sorry there was a typo in my previous email. PKG_CONFIG_PATH has to be set to the location of "libcurl.pc", which is inside lib/pkgconfig. So the full code is this: brew install curl-openssl pkgconfig curl -OL https://github.com/curl/curl/raw/master/docs/examples/crawler.c export PKG_CONFIG_PATH="/usr/local/opt/curl-openssl/lib/pkgconfig" cc -o crawler crawler.c $(pkg-config --cflags --libs libxml-2.0 libcurl) Use otool -L to confirm that executable links against the correct libcurl shared library: otool -L crawler # crawler: # /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0) # /usr/local/opt/curl-openssl/lib/libcurl.4.dylib (compatibility version 11.0.0, current version 11.0.0) # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1) --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libcurl - MacOS 10.14 ...
On Tue, Nov 12, 2019 at 8:51 PM Daniel Stenberg via curl-library wrote: > > On Tue, 12 Nov 2019, Bryan O'Brien via curl-library wrote: > > > Courtesy of curl_version(), heres the version I’m using > > > > libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1 > > > > If anyone is aware of a more recent version, or can point me towards some > > newbie build instructions, I’d appreciate it. > > You build and install libcurl the same way on mac as on any *nix: > > $ ./configure > $ make > $ make test (optional) > $ make install > > For more details, see https://curl.haxx.se/docs/install.html > > There are also package managers for mac as well, like homebrew, but it's not > clear to be if they offer libcurl installs (like with headers etc). Yes they do. There are two versions: one build with macos secure transport, and one build with openssl+nghttp2: - https://github.com/homebrew/homebrew-core/blob/master/Formula/curl.rb - https://github.com/homebrew/homebrew-core/blob/master/Formula/curl-openssl.rb Because this homebrew package would conflict with a native macos version of curl, it is not automatically linked into /usr/local, so you need to set PKG_CONFIG_PATH. For example use this to build the crawler example against the latest curl with http2 support: brew install curl-openssl pkgconfig curl -OL https://github.com/curl/curl/raw/master/docs/examples/crawler.c export PKG_CONFIG_PATH="/usr/local/opt/curl-openssl" cc -o crawler crawler.c $(pkg-config --cflags --libs libxml-2.0 libcurl) ./crawler --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libcurl - MacOS 10.14 ...
On Tue, 12 Nov 2019, Bryan O'Brien via curl-library wrote: Courtesy of curl_version(), heres the version I’m using libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1 If anyone is aware of a more recent version, or can point me towards some newbie build instructions, I’d appreciate it. You build and install libcurl the same way on mac as on any *nix: $ ./configure $ make $ make test (optional) $ make install For more details, see https://curl.haxx.se/docs/install.html There are also package managers for mac as well, like homebrew, but it's not clear to be if they offer libcurl installs (like with headers etc). -- / daniel.haxx.se | Get the best commercial curl support there is - from me | Private help, bug fixes, support, ports, new features | https://www.wolfssl.com/contact/--- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
libcurl - MacOS 10.14 ...
Hi Folks, Courtesy of curl_version(), heres the version I’m using libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1 If anyone is aware of a more recent version, or can point me towards some newbie build instructions, I’d appreciate it. Best Regards, Bryan smime.p7s Description: S/MIME cryptographic signature --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: reproducible, but limited, SEGFAULTS after upgrade curl/libcurl 7.66 -> 7.67. curl issue, or not?
>> *Is* this^^ likely a curl issue? > > > There's not enough information to tell. I downloaded that appdata file > without a problem using 7.67 but I'm using Windows. We need a link to > the issue filed with the repo and a minimal self-contained example that > can be used to reproduce. Find out if there's a way to enable curl > verbosity, like for example in git you enable it with environment > GIT_CURL_VERBOSE, maybe zypper has something similar. Got it. Will see what/if I can cobble up ... --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On Tue, 12 Nov 2019, Niall.oReilly+lists--- via curl-library wrote: The very long name (`const char *bad = "this.is.` _et seq_.) is invalid according to RFCs 1034 and 1035 both because it is too long and because it contains embedded zero-length labels (which are represented by consecutive dots). A strict encoder should reject it early, and so frustrate the attempt to force a buffer overflow. Ah! We should probably A) fix that and refuse such names with zero labels and B) update the used host names in the test... A validly encoded pure (without EDNS) single query must fit in 272 (12 + 256 + 4) octets, which is well below the buffer size of 512 provided in `struct dnsprobe`. Then maybe that's an additional fix if we have no other reason for a larger buffer! -- / daniel.haxx.se | Get the best commercial curl support there is - from me | Private help, bug fixes, support, ports, new features | https://www.wolfssl.com/contact/ --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: reproducible, but limited, SEGFAULTS after upgrade curl/libcurl 7.66 -> 7.67. curl issue, or not?
On 11/12/2019 11:38 AM, PGNet Dev via curl-library wrote: Retrieving:http://download.opensuse.org/distribution/leap/15.1/repo/oss/repodata/57cb86253c19b7c80ce4de1fb5a081a842e24eac5308c84b159d6728e3edc183-appdata.xml.gz -[starting]Segmentation fault (core dumped) ... ``` journal reports, ``` journalctl -f ... Nov 11 13:37:51 prod223 kernel: [13139.140228] zypper[26689]: segfault at 6d6172617088 ip 7fbced4c5961 sp 7ffdf070a3e0 error 4 in libcurl.so.4.6.0[7fbced488000+88000] Nov 11 13:37:51 prod223 kernel: [13139.140236] Code: 8c 94 00 00 00 39 71 28 0f 8c 8b 00 00 00 0f 1f 00 49 89 41 08 49 89 c1 48 8b 40 08 48 39 78 20 7e bd 48 8b 10 48 85 d2 74 3f <48> 39 7a 20 7f 19 7c 05 39 72 28 7f 12 48 89 c2 49 89 10 49 89 d0 Nov 11 13:37:51 prod223 kernel: [13139.140245] RIP: 0033:0x7fbced4c5961 Nov 11 13:37:51 prod223 kernel: [13139.140246] Code: 8c 94 00 00 00 39 71 28 0f 8c 8b 00 00 00 0f 1f 00 49 89 41 08 49 89 c1 48 8b 40 08 48 39 78 20 7e bd 48 8b 10 48 85 d2 74 3f <48> 39 7a 20 7f 19 7c 05 39 72 28 7f 12 48 89 c2 49 89 10 49 89 d0 Nov 11 13:37:51 prod223 kernel: [13139.140247] RSP: 002b:7ffdf070a3e0 EFLAGS: 00010202 Nov 11 13:37:51 prod223 kernel: [13139.140248] RAX: 55e3802b0da8 RBX: 55e380270938 RCX: 55e380270938 Nov 11 13:37:51 prod223 kernel: [13139.140249] RDX: 6d6172617068 RSI: 00053d91 RDI: 3353 Nov 11 13:37:51 prod223 kernel: [13139.140250] RBP: 3353 R08: 7ffdf070a3e0 R09: 7ffdf070a3e0 Nov 11 13:37:51 prod223 kernel: [13139.140251] R10: 7ffdf070a420 R11: 3353 R12: 00053d91 Nov 11 13:37:51 prod223 kernel: [13139.140251] R13: 55e380270938 R14: 55e380218890 R15: 55e380270ae0 Nov 11 13:37:51 prod223 kernel: [13139.140252] FS: 7fbcef7caa00 GS: ... ``` Again, downgrading curl/libcurl -> v7.60 fixes the problem. Also, WITH v7.67, *DIS*abling the one repo also fixes the problem. I'm not sure this*is* curl, vs something else systemic ... Can provide add'l info as needed. *Is* this^^ likely a curl issue? There's not enough information to tell. I downloaded that appdata file without a problem using 7.67 but I'm using Windows. We need a link to the issue filed with the repo and a minimal self-contained example that can be used to reproduce. Find out if there's a way to enable curl verbosity, like for example in git you enable it with environment GIT_CURL_VERBOSE, maybe zypper has something similar. --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
reproducible, but limited, SEGFAULTS after upgrade curl/libcurl 7.66 -> 7.67. curl issue, or not?
I recently upgraded distro-packaged (openSUSE) curl/libcurl from v7.66 -> 7.67. Subsequently, when I exec the distro's package management tool (zypper), it works for accessing ALL defined repos without problem/error, except for one, at which it reproducibly SEGFAULTs (~60+ machines, so far). Downgrade from from v7.67 -> 7.60 fixes the problem. (7.66 pkgs vanished from distro repos ... I _suspect_ 7.66 pkgs would be OK, unless some _other_ change is causal ...) distro suggests it's a curl problem, not zypper. I'm stymied atm as to why/how a SEGFAULT is dependent on accessing a single repo. (something in the content, or size, of the payload?) Here's what I see on multiple machines: ``` zypper -vv up ... Checking whether to refresh metadata for OS15-non-oss Retrieving: repomd.xml [done] Retrieving: media .[done] Checking whether to refresh metadata for OS15-oss Retrieving: http://download.opensuse.org/distribution/leap/15.1/repo/oss/media.1/media [done] Retrieving: http://download.opensuse.org/distribution/leap/15.1/repo/oss/repodata/repomd.xml.asc ..[done] Retrieving: http://download.opensuse.org/distribution/leap/15.1/repo/oss/repodata/repomd.xml.key [done (988 B/s)] Retrieving: http://download.opensuse.org/distribution/leap/15.1/repo/oss/repodata/repomd.xml ..[done] Repository: OS15-oss Key Name: openSUSE Project Signing Key Key Fingerprint: 22C07BA5 34178CD0 2EFE22AA B88B2FD4 3DBDC284 Key Created: Mon 05 May 2014 01:37:40 AM PDT Key Expires: Thu 02 May 2024 01:37:40 AM PDT Rpm Name: gpg-pubkey-3dbdc284-53674dd4 Retrieving: http://download.opensuse.org/distribution/leap/15.1/repo/oss/repodata/12756b0bcbe274b3bdf178191baea85635c717e3321644fb2ca8807e36be87fa-primary.xml.gz .[done] Retrieving: http://download.opensuse.org/distribution/leap/15.1/repo/oss/repodata/57cb86253c19b7c80ce4de1fb5a081a842e24eac5308c84b159d6728e3edc183-appdata.xml.gz -[starting]Segmentation fault (core dumped) ... ``` journal reports, ``` journalctl -f ... Nov 11 13:37:51 prod223 kernel: [13139.140228] zypper[26689]: segfault at 6d6172617088 ip 7fbced4c5961 sp 7ffdf070a3e0 error 4 in libcurl.so.4.6.0[7fbced488000+88000] Nov 11 13:37:51 prod223 kernel: [13139.140236] Code: 8c 94 00 00 00 39 71 28 0f 8c 8b 00 00 00 0f 1f 00 49 89 41 08 49 89 c1 48 8b 40 08 48 39 78 20 7e bd 48 8b 10 48 85 d2 74 3f <48> 39 7a 20 7f 19 7c 05 39 72 28 7f 12 48 89 c2 49 89 10 49 89 d0 Nov 11 13:37:51 prod223 kernel: [13139.140245] RIP: 0033:0x7fbced4c5961 Nov 11 13:37:51 prod223 kernel: [13139.140246] Code: 8c 94 00 00 00 39 71 28 0f 8c 8b 00 00 00 0f 1f 00 49 89 41 08 49 89 c1 48 8b 40 08 48 39 78 20 7e bd 48 8b 10 48 85 d2 74 3f <48> 39 7a 20 7f 19 7c 05 39 72 28 7f 12 48 89 c2 49 89 10 49 89 d0 Nov 11 13:37:51 prod223 kernel: [13139.140247] RSP: 002b:7ffdf070a3e0 EFLAGS: 00010202 Nov 11 13:37:51 prod223 kernel: [13139.140248] RAX: 55e3802b0da8 RBX: 55e380270938 RCX: 55e380270938 Nov 11 13:37:51 prod223 kernel: [13139.140249] RDX: 6d6172617068 RSI: 00053d91 RDI: 3353 Nov 11 13:37:51 prod223 kernel: [13139.140250] RBP: 3353 R08: 7ffdf070a3e0 R09: 7ffdf070a3e0 Nov 11 13:37:51 prod223 kernel: [13139.140251] R10: 7ffdf070a420 R11: 3353 R12: 00053d91 Nov 11 13:37:51 prod223 kernel: [13139.140251] R13: 55e380270938 R14: 55e380218890 R15: 55e380270ae0 Nov 11 13:37:51 prod223 kernel: [13139.140252] FS: 7fbcef7caa00 GS: ... ``` Again, downgrading curl/libcurl -> v7.60 fixes the problem. Also, WITH v7.67, *DIS*abling the one repo also fixes the problem. I'm not sure this *is* curl, vs something else systemic ... Can provide add'l info as needed. *Is* this^^ li
Unit test 1655
Hi. [I’ve had a lot of trouble sending this. Apologies in case multiple copies arrive on the list.] I’m looking to extend `dohprobe()` to allow retrieval of other host attributes than addresses, some of which are published at prefixed DNS names relative to the host name. I have had to take account of unit test 1655, which puzzles me. I’m wondering whether it may be appropriate to open an issue. The very long name (`const char *bad = "this.is.` _et seq_.) is invalid according to RFCs 1034 and 1035 both because it is too long and because it contains embedded zero-length labels (which are represented by consecutive dots). A strict encoder should reject it early, and so frustrate the attempt to force a buffer overflow. A validly encoded pure (without EDNS) single query must fit in 272 (12 + 256 + 4) octets, which is well below the buffer size of 512 provided in `struct dnsprobe`. Thanks in anticipation for advice on how best to proceed. Niall O’Reilly--- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Feature idea: wakeup option for curl_multi_wait/poll
I just realized this feature request is already tracked under the https://github.com/curl/curl/wiki/curl_multi_unblock wiki page. It says on the page that some kind of locking (mutexes) are necessary to implement this, I don't think that's true. As long as this unblock function is only accessing the write half of the socketpair and only does an swrite() call, it should be thread-safe (assuming the application had some kind of synchronization between the threads after setting up the multi and before this unblock function is called so the fd itself is seen correctly by the other thread; the send() syscall itself is thread-safe). It's also the app's responsibility not to call unblock after destructing the multi, but this should be like this either way. From: curl-library on behalf of Gergely Nagy via curl-library Sent: Tuesday, November 12, 2019 9:50 AM To: curl-library@cool.haxx.se Cc: Gergely Nagy Subject: Feature idea: wakeup option for curl_multi_wait/poll Hi! We have a use-case where we use a single curl multi with lots of easy connections which are added/removed in a relatively adhoc manner, anytime. We previously used the multi_socket API with libuv for this, but we'd like to migrate to the curl_multi_poll API. The problem is that when we for example want to add a new easy connection, we have to wait until curl_multi_poll wakes up. We could play with setting the timeout low enough so this waiting time is not too long, but not too low to keep CPU usage low. We found a solution to this, where we create a socketpair (or emulate one on Windows just like Curl_socketpair does), and add one of the ends as an extra fd. This way we can wake the thread up by writing to the other end. This works pretty well, but I think this could be a feature of the multi API inside curl, probably useful for others as well. As the Curl_socketpair function is not public, doing this manually includes copying code as well. Idea about how this could be added to the API: 1. Two fds should be added to the Curl_multi struct left in an invalid state at first (so there won't be a perf penalty for those not wanting to use this feature) 2. There should be a new CURLMOPT_ENABLE_WAKEUP option that initializes a socketpair and stores the fds when set to 1L. (if some platform doesn't support socketpair at all, it could return an errorcode (for example CURLM_BAD_SOCKET, CURLM_UNKNOWN_OPTION or some new error code just added for this), which could be handled by applications for example by lowering the timeout to make the unfortunately necessary waits shorter) 3. There should be a new function curl_multi_wakeup which writes one byte to the writing end of the socketpair. 4. If the option was set successfully, Curl_multi_wait should add the reading end of the socketpair to its fd list, and after the actual Curl_poll call, if this fd is marked POLLIN, then it should try to read out all data from it and discard it. (This is not so trivial, because it should avoid reading in a long loop if there are lots of wakeup calls meanwhile). Would you be interested in supporting this feature? Thanks, Gergely Nagy --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Feature idea: wakeup option for curl_multi_wait/poll
Hi! We have a use-case where we use a single curl multi with lots of easy connections which are added/removed in a relatively adhoc manner, anytime. We previously used the multi_socket API with libuv for this, but we'd like to migrate to the curl_multi_poll API. The problem is that when we for example want to add a new easy connection, we have to wait until curl_multi_poll wakes up. We could play with setting the timeout low enough so this waiting time is not too long, but not too low to keep CPU usage low. We found a solution to this, where we create a socketpair (or emulate one on Windows just like Curl_socketpair does), and add one of the ends as an extra fd. This way we can wake the thread up by writing to the other end. This works pretty well, but I think this could be a feature of the multi API inside curl, probably useful for others as well. As the Curl_socketpair function is not public, doing this manually includes copying code as well. Idea about how this could be added to the API: 1. Two fds should be added to the Curl_multi struct left in an invalid state at first (so there won't be a perf penalty for those not wanting to use this feature) 2. There should be a new CURLMOPT_ENABLE_WAKEUP option that initializes a socketpair and stores the fds when set to 1L. (if some platform doesn't support socketpair at all, it could return an errorcode (for example CURLM_BAD_SOCKET, CURLM_UNKNOWN_OPTION or some new error code just added for this), which could be handled by applications for example by lowering the timeout to make the unfortunately necessary waits shorter) 3. There should be a new function curl_multi_wakeup which writes one byte to the writing end of the socketpair. 4. If the option was set successfully, Curl_multi_wait should add the reading end of the socketpair to its fd list, and after the actual Curl_poll call, if this fd is marked POLLIN, then it should try to read out all data from it and discard it. (This is not so trivial, because it should avoid reading in a long loop if there are lots of wakeup calls meanwhile). Would you be interested in supporting this feature? Thanks, Gergely Nagy --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
