Re: localhost to be truly local?
FWIW, and with the caveat that I am not a **real** DNS expert. On 10 May 2021, at 14:52, Daniel Stenberg via curl-library wrote: I've created PR #7039 that makes "localhost" resolve to 127.0.0.1 and ::1 without using the resolver [1]. The point of this is to make sure localhost is the local host for sure. I think it makes sense to move [[RFC6761][]] compliance inside the application. curl's --resolve option and its libcurl counterpart still allows a user to make localhost URL's connect to another IP address, just like for any other name. Compliance with [[RFC6761]] should surely have priority over consistency. 8-) Your feedback and thoughts on this are most welcome! €0,02 /Niall [RFC6761]: https://datatracker.ietf.org/doc/html/rfc6761#section-6.3 --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
POC for ECH support in curl and libcurl
Hello. I would like to let people know of a proof-of-concept implementation of ECH in curl and libcurl. This uses OpenSSL as back-end and interoperates with Cloudflare’s demonstration server. For more information, please see https://github.com/niallor/curl/blob/ECH-WIP/docs/ECH.md Best regards. Niall O’Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
ECH reset
Hi. Now that IETF work on ECH and service binding seems to be converging, I’ve started looking at how to refactor the experimental ESNI code from nearly a year ago. I’ld like to start by to updating docs/ECH.md. Please let me know whether https://github.com/niallor/curl/blob/ECH-reset/docs/ECH.md seems “PR-ready”. Best regards, Niall O’Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
ESNI, Encrypted Client Hello, DNS developments
Hi. Work I was involved in to add ESNI support to libcurl was suspended quite a number of months ago as the IETF TLS WG decided on, and since specified, a different approach. After dealing with some other distractions, I expect to pick this up again soonish, but most likely not before the new year. Here is a summary of the new situation. - ESNI is no longer an independent feature, but an element of Encrypted Client Hello (ECHO); - SVCB and HTTPS records have been introduced in the DNS for binding (alternative sets of) service parameters to a hostname; - To support ECHO, an application will need to look for SVCB or HTTPS RRs, not just A and RRs; - POSIX getaddrinfo() only provides data from A and RRs, so will no longer be adequate. For those who are interested, here is a link to a presentation explaining the SVCB and HTTPS resource records, which was given at an interim virtual meeting of the RIPE DNS Working Group early in October; it has only recently become available on the RIPE website. https://www.ripe.net/participate/ripe/wg/active-wg/dns/remote-sessions/svcb_https_-ripe-2020.pdf Best regards, Niall O’Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: At 15,000 commits
Helt sent, men grattis! On 18 Sep 2020, at 21:41, Daniel Stenberg via curl-library wrote: It's a meaningless number, but I just reached 15,000 commits in the curl master branch today and I figured it could be fun to celebrate so I blogged a little about it: --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
FYI Tentative ECH roadmap
Feedback welcome: https://github.com/niallor/curl/commit/873a5053c830fe68fa278e1b65cf4a7a0971ecce Thanks in anticipation Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: We got the gold badge!
On 13 Jun 2020, at 21:56, Daniel Stenberg via curl-library wrote: > As of just a few moments ago we fulfill the gold level best practices! Grattis! --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: CII Best Practices: curl now at Silver level
On 12 Jun 2020, at 10:41, Daniel Stenberg via curl-library wrote: "The project MUST clearly identify small tasks that can be performed by new or casual contributors" ... as this then seems to be *the only* remaining critera for curl reaching gold level CII Best Practices, I'm tempted to do something about it. I haven't worked out exactly what or how. I don't know what motivates this requirement, or how it may have been coloured by an assumption of uniformity of project leadership/management style across the universe of projects. There seems to be an underlying idea that the "small tasks" can be identified in advance. It ain't necessarily so. It may well be that a casual contributor is better able than the core team to identify a previously unconsidered work item that fits the "small task" description, simply on account of having a different perspective. I'm just such a casual contributor. Some of the results of small tasks I've performed were deemed useful and committed to the code; others, inappropriate or not timely. That's all fine. Checking the spelling (as suggested by Jeffrey Walton), or the grammar, or the clarity of the documentation is always a set of unfinished tasks, each of them small, each a perfect fit to the requirement. I think it would send a very positive message about the culture of this project to include some explicit statement, encouraging a casual reviewer, whose experience equips them to identify a potential improvement here or there, to engage by proposing their own "small tasks". From my experience, the encouraging culture is already there. Don't be shy about documenting it: go for gold! Niall O'Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Repeated calls to ares_init when using c-ares
On 29 Jan 2020, at 7:51, Daniel Stenberg via curl-library wrote: 2. A simpler method would possibly be to use a single c-ares "channel" for a whole multi handle, as then you would just make sure to keep the multi handle around when doing multiple transfers and c-ares wouldn't have to re-init. This seems to be the kind of approach which would enable use of c-ares for fetching host attributes other than addresses, such as might be carried by HTTPSSVC. Niall O'Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Tests and randomness
On 19 Dec 2019, at 9:27, Daniel Stenberg via curl-library wrote: > Thoughts? All good, especially giving the seed some, but limited, persistence. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Funding curl up attendance
On 27 Nov 2019, at 10:03, Daniel Gustafsson via curl-library wrote: > I'm not sure if we need to make it more explicit, but if we make funding > students in some way different then maybe we should add that a proof of > enrollment is required? Maybe RIPE's RACI programme could be a model. https://www.ripe.net/participate/ripe/raci /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
DOH: minor bug: store_cname() is storing duplicate CNAMEs
I’m not sure whether this is worth opening an issue. If the target hostname is an alias, the DNS returns a CNAME in the answer to each of the queries which Curl_doh() launches. In the standard libcurl build, there are two of these, with QTYPE of A and respectively. During decoding, these are found and stored separately, although the values are identical. I (or somebody else) may find time at some stage to add duplicate-suppression to store_cname(). Here below is a snippet from the verbose progress message stream. /Niall ``` * a DOH request is completed, 0 to go * DOH Host name: www.jamm.ie * TTL: 3150 seconds * DOH A: 185.24.233.219 * DOH : 2a04:2e00:0001:0001::::000a * CNAME: vps.jamm.ie * CNAME: vps.jamm.ie * Trying 185.24.233.219:443... ``` — Ends —--- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
DOH: design notes: gathering additional host attributes
Hello.
A host’s IPv4 or IPv6 address is needed in order to establish a connection.
If DOH is enabled, the function *Curl_doh()* is used to retrieve these
from the DNS. *Curl_doh()* has two dedicated “probe slots” for holding DOH
query state independently for each of the DNS QTYPEs, A and .
In some use cases (such as ESNI), additional host attributes, which may be
available from the DNS, are needed as connection parameters. Defining an
additional “probe slot” for each such attribute seems the simplest way to
hold the corresponding query state.
In order to avoid collisions where a slot is inadvertently used for more
than one purpose, it may be useful to assign symbolic names to the slot
positions. Defining these conditionally at build-time will obviate
keeping yet another registry of code-points in some header file.
The code snippets below show how I’m thinking of doing this.
/Niall
```
/* in lib/urldata.h */
/* ... */
enum doh_slots {
/* Explicit values for first two symbols so as to match hard-coded
* constants in existing code
*/
DOH_PROBE_SLOT_IPADDR_V4 = 0, /* make 'V4' stand out for readability */
DOH_PROBE_SLOT_IPADDR_V6 = 1, /* 'V6' likewise */
/* Space here for (possibly build-specific) additional slot definitions */
#ifdef WANT_DOH_FOOBAR_TXT
DOH_PROBE_SLOT_FOOBAR_TXT,/* for example */
#endif
#ifdef WANT_DOH_FOOBAR_SVCB
DOH_PROBE_SLOT_FOOBAR_SVCB, /* another example */
#endif
/* AFTER all slot definitions, establish how many we have */
DOH_PROBE_SLOTS
};
/* ... */
struct dohdata {
struct curl_slist *headers;
struct dnsprobe probe[DOH_PROBE_SLOTS];
unsigned int pending; /* still outstanding requests */
const char *host;
int port;
};
/* ... */
```
— Ends —---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: DOH: retrieving an RRset from a child node
On 20 Nov 2019, at 14:10, Daniel Stenberg wrote: SVCB is what the current ESNI draft wants, right? (btw there's now a poll going for what to name those new records over at: https://lists.w3.org/Archives/Public/ietf-http-wg/2019OctDec/0117.html ) IIUC, SVCB will be published at a prefix node, and HTTPSSVC is intended to be a special flavour of SVCB which will be published at the node corresponding to the hostname. Seems straight forward and sensible to me! I just submitted a PR. The big half of the effort was for the unit test. The TXT records for old-draft- ESNI that cloudflare will soon stop providing? For some value of “soon”. Until we have names and code-points agreed for the new QTYPEs, TXT seems worth working with. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
DOH: retrieving an RRset from a child node
Hello. Some data is published in the DNS at a child node of the node which corresponds to the hostname, as documented in [RFC8552](https://datatracker.ietf.org/doc/rfc8552/). In particular, current (experimental) ESNI deployment uses TXT records and child-prefix "_esni". I've been thinking about how to extend dohprobe() to query for an RRset containing such data, which action may be useful for other use cases involving existing TXT, SRV, TLSA, and URI RR types, as well as future RRtypes, such as SVCB. Here's what I'm minded to do. - Introduce new functions dohprobepfx() and doh_encodepfx(), corresponding to dohprobe() and doh_encode respectively, each with an additional `const char *prefix` parameter. - Retain dohprobe() and doh_encode(), but as trivial wrappers around the "pfx" functions, each passing NULL as the argument to the corresponding "pfx" function, so that existing invoking code need not be modified, and duplication of code can be avoided. If this doesn't seem too daft, I'll work up a PR. I expect to deal with reception of the TXT response in a separate PR, so as to keep changes small and modular. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On 13 Nov 2019, at 9:36, Daniel Stenberg wrote: It might be worth splitting out and fixing in a separate pull-request, sure! OK, done. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On 13 Nov 2019, at 13:41, Daniel Stenberg wrote: Is there a reason the DoH code can't just use with exactly the host name it gets passed, using a trailing dot or not? That’s pretty-well the answer I was looking for. I’m not aware of a compelling reason that it shouldn’t be tolerant, just as you suggest, but wanted to be sure that I wasn’t missing something. Whether the trailing dot is present or not has to be taken into account if an exact estimate (rather than a safe over-estimate) of the required buffer size (constant `expected_len` in `doh_encode()`) is required. For example, `“example.com”` and `“example.com.”`, although of differing lengths as strings, are both encoded as the same QNAME `(6)example(3)com(0)`. For accommodating corner cases, and for keeping the unit test as simple as possible, I think I’ll add some complexity into the estimation of `expected_len`. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On 13 Nov 2019, at 13:33, Daniel Stenberg wrote: Why twice? If it is a generic fix to DoH in curl, that could be lifted out from your other work and merged indepdendently. Once. That’s pretty-well ready, including prefix support, which I’m not minded to cut out, only to have to add it back in at a later stage. Prefix support will be useful independently of ESNI for SRV and other use cases which you identified in an earlier thread. The ESNI work is in “another part of the forest”. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On 13 Nov 2019, at 9:36, Daniel Stenberg wrote: I think it primarily needs a valid input name (no zero length labels) that is longer than the given output buffer, as that test tries to verify that the boundary checks for that are fine and causes no overwrite. There’s a notational ambiguity which needs a decision. Have you thought about the trailing dot? If so, do you have a clear idea (or even strong feelings) about whether it should be either tolerated or forbidden? It can’t be required, as doing so would introduce pervasive conflict with how libcurl already handles a hostname parsed from a URL. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On 13 Nov 2019, at 9:36, Daniel Stenberg wrote: It might be worth splitting out and fixing in a separate pull-request, sure! Hmm. I’m not sure what you mean here. If you mean a PR which covers RFC1035 compliance but not prefix support, I would find myself doing almost the same work twice. The prospect doesn’t fill me with enthusiasm. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Unit test 1655
On 12 Nov 2019, at 18:51, Daniel Stenberg wrote: Ah! We should probably A) fix that and refuse such names with zero labels and B) update the used host names in the test... I think I’ve covered item A as a side effect while working on prefix support. https://github.com/niallor/curl/commit/228633becd613b4c9e329117bb20d850f6418c8a Is this worth a PR yet? Or an issue? For item B, I think a more elaborate test will be needed. I’m not sure what’s needed. /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Managing application data fetched from DNS (eg for ESNI)
Hi. After too many attempts to follow up Daniel's (off-list) reaction to my earlier mail, here's a quick summary of how I think the questions raised can most simply be resolved. On 11 Sep 2019, at 16:33, niall.oreilly+li...@ucd.ie wrote: > * Extend `struct dohdata` or use a different > structure for tracking DoH transactions whose > purpose is other than address resolution. Extend dohdata. > * If extending `struct dohdata`, dedicate an > additional slot per QTYPE, or introduce an > array of ‘generic’ slots. Dedicate additional slots, perhaps per (data-type, origin) rather than per QTYPE, as some QTYPEs are re-used for different purposes using distinct QNAME prefixes. More thought seems needed here. I'm minded to prototype a solution and have feedback on its appropriateness. > * Choose a structure for caching returned > data, as existing (hostname, port, address) > tuples cannot represent all the significant > data items (QNAME prefix, QTYPE). Rather than a shared cache as used at present for address (A/) data, a per-request or per-easy data structure seems good for a first attempt. > * Decide what DNS-derived data must be present > before advancing the connection state machine. Existing behaviour is to wait for all launched dohprobe transactions to complete, even though (modulo protocol-specific reachability) either an IPv4 or an IPv6 address should be sufficient. Looking ahead to ESNI, relevant key data might be published either as TXT with prefix or as TYPE65439 (pending standardization) without prefix, so two transactions have to be launched. Waiting for all to complete, and postponing optimization, as currently, seems the right next step at this stage. Best regards Niall O'Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Apologies for possible duplicate postings
Hi. Some messages which I sent to this list earlier in the week didn't arrive. Both Daniel Stenberg and I are puzzled. I'm trying to identify differences at source between these messages and one which arrived as expected yesterday. As I find things that I can adjust, I'm doing this and re-sending until one reaches its destination. If I misjudge the latencies, duplicates may arrive. I'm sorry for the noise. Best regards Niall O'Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
doh_encode() in lib/doh.c truncates DNS QTYPE to low 8 bits
PR on the way to allow full range [0..65535] for DNS QTYPE in doh_encode(). /Niall --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Update on ESNI initiative
Hi. On 31 May 2019, at 9:52, Niall O'Reilly wrote: More will follow in due course. So far, we have the network side, between **curl** client and an ESNI-aware server, working. Instructions for trying this out are here: [https://github.com/niallor/curl/blob/master/ESNI-README.md](https://github.com/niallor/curl/blob/master/ESNI-README.md). In the **curl** app, we’re cheating by providing parameters for ESNI as command-line options instead of fetching what’s published in the DNS. We’ll be working on that next. Niall O’Reilly --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Fwd: ESNI initiative
Apologies for my mistake earlier: I sent this from an address which isn't allowed to send to this list. Forwarded message: From: Niall O'Reilly To: curl-library@cool.haxx.se Subject: ESNI initiative Date: Wed, 29 May 2019 15:03:21 +0100 Hello. I'm not sure whether posting to this list is the right thing to do; apologies if not. I'm looking for feedback on what I suggest below. The DEfO project (https://defo.ie) has work in progress to integrate prototype OpenSSL support for ESNI into a number of applications. Not surprisingly, curl is high on our list. We're still exploring how the curl application and library are structured and built. One of us will say more when we have something presentable. Just now, there are a couple of things which I think it would be timely to try to agree with people outside the DEfO project, specifically a feature-bit allocation and a pre-processor symbol for protecting conditional-compilation code sections. Following the existing pattern for the `ALTSVC` feature, these might be (for example), ``` #define CURL_VERSION_ESNI (1<<25) /* ESNI support */ ``` and `USE_ESNI`. If this is premature, or doesn't make sense, I'ld appreciate being told. Feedback is not necessarily confirmation. Thanks in anticipation of a reaction, and best regards, Niall O'Reilly Tolerant Networks Ltd. --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
