HTTP/2 server push, callback?
Hi again,
We're in feature-freeze mode for this pending release but that will end with a
release in three weeks and then we'll be open for new features again. I
thought I'd throw some server push ideas in the mean time and see if anyone
has ideas or improvements for them!
I imagine two new options:
- CURLMOPT_PUSHFUNCTION [callback]
CURLMOPT_PUSHDATA [callback user pointer]
These are two new options for curl_multi_setopt()
Without the CURLMOPT_PUSHFUNCTION set, pushed streams will be refused.
This is how the callback would work:
int curl_stream_push(CURL *newhandle,
unsigned int stream_id,
struct curl_push *headers,
void *userp);
This callback gets called when a new stream is being pushed by the
server. The 'newhandle' is a newly created easy handle that represents this
new transfer. This handle is otherwise naked and will need a suitable set of
options set before it will work the way the application wants it to.
If the callback returns positively, the new easy handle will be added to the
multi handle automatically.
struct curl_push {
const char *method;/* example: GET */
const char *path; /* example: /download-154/image.png */
const char *scheme;/* example: https */
const char *authority; /* example: www.example.com:443 */
};
The callback returns one of:
CURL_PUSH_OK - we have accepted the stream and it can now start receiving
data, the ownership of the CURL handle has been taken over by
the application.
CURL_PUSH_DENY - the callback denies the stream and no data for this will
reach the application, the easy handle will be destroyed by
libcurl
* - all other return codes are reserved for future use
Thoughts?
--
/ daniel.haxx.se
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Connecting to multiple hosts that have the same hostname
Hi, I have a challenging scenario for libcurl. I want to connect with https to a cluster of ADFS hosts (Active Directory Federation Services). These hosts have the same hostname, but different IP addresses. For example: - host.example.org, IP: 10.0.0.1 - host.example.org, IP: 10.0.0.2 When establishing a connection, I want to control to which host libcurl should connect to (10.0.0.1 or 10.0.0.2) because the hosts have separate user sessions. Furthermore ADFS needs SNI, so it is necessary to use the hostname in the URL (https://host.example.org). It is not possible to just use the IP address (https://10.0.0.1/ or https://10.0.0.2/). As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: - CURLOPT_RESOLVE is not a local setting because it pre-populates the DNS cache. All easy handles that use the same multi handle share a DNS cache. Setting different IP addresses for the same hostname using CURLOPT_RESOLVE may therefore lead to race conditions. - A wrong existing connection may get reused. libcurl only looks at the hostname, and does not remember that the connection has been set up using a special IP address with CURLOPT_RESOLVE. (I have not tested this, but I have looked at libcurl's source code.) It is possible to work around these problems by using CURLOPT_RESOLVE with an artificial hostname in the URL, e.g. https://host.example.org-10.0.0.1/; - but then the server drops the connection, because the artificial hostname is also used for SNI. A libcurl option to control the SNI hostname (and also the hostname used for the server certificate checks) would make this workaround viable. A clean solution needs to address the problems mentioned above: - CURLOPT_RESOLVE should only affect a single easy handle - A new option to control the connection cache's behavior (reuse based on hostname only or reuse based on hostname + IP address). This may be implemented in a more general way with a label (string) that gets attached to a connection, and libcurl is only allowed to reuse connections when the labels match. What do you think? All ideas and suggestions are welcome. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Question regarding multiple IMAP operations
Ray, I've actually written C89 code to parse IMAP responses. libcurl IMAP
is indeed garbage, because the only thing it does for you that a straight
socket doesn't is encryption (which is well documented w/tons of code
samples anyway). So as soon as I remove these crashes I will post my code
to hopefully make it a bit more of a complete IMAP library. I've already
come this far w/libcurl despite it's awkward callback schemes so don't feel
like switching now.
Let's say I want first list folders, the default, then 1 second later I
want to EXAMINE OUTBOX, then 1 second later I want to read first message
imap://imap.example.com/INBOX/;UID=1 There are no code samples showing
how to do multiple requests like this.
Would something like this be the correct way? Or is this what the multi_
function families are for? Not this exactly, but doing stuff like this I'm
getting a lot of different crashes, and I'm not using any handles after
calling cleanup.
Please specify if this is the intended usage, or if I need to be doing
something else, thanks.
int main(void)
{
CURL *curl;
CURLcode res = CURLE_OK;
curl = curl_easy_init();
if(curl) {
/* Set username and password */
curl_easy_setopt(curl, CURLOPT_USERNAME, user);
curl_easy_setopt(curl, CURLOPT_PASSWORD, secret);
/* This will fetch message 1 from the user's inbox */
curl_easy_setopt(curl, CURLOPT_URL, imap://
imap.example.com/INBOX/;UID=1);
/* Perform the fetch */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, curl_easy_perform() failed: %s\n,
curl_easy_strerror(res));
curl_easy_setopt(curl, CURLOPT_URL, EXAMINE OUTBOX);
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
printf(fuck\n);
/* Always cleanup */
curl_easy_cleanup(curl);
}
return (int)res;
}
On Wed, May 27, 2015 at 3:33 AM, Ray Satiro via curl-library
curl-library@cool.haxx.se wrote:
On 5/26/2015 9:31 AM, Adam wrote:
What is the proper way to send multiple IMAP commands? For example, first
I'd like to get a listing of all the folder names, then I want to list the
unread files in the folder names.
I've tried using the same CURL* curl handle, sometimes it crashes on
curl_easy_cleanup() (single threaded app), and sometimes it crashes on the
second call to curl_easy_perform(). It depends on the IMAP server, gmail
causes a crash, hotmail doesn't. I've even tried opening/closing a new
handle for each operation and this causes crashes also.
What is the proper way to do this? There are no code samples for this on
the libcurl homepage. Thanks.
The proper way is multiple requests and don't call curl_easy_cleanup until
after you're done with the handle. You want to reuse the handle until then.
Please note as mentioned in the curl_easy_cleanup doc that Any use of the
handle after this function has been called and have returned, is illegal.
[1]
So you have custom request(s) to get the unread ids then you get the
subjects. Getting the subjects from what I remember is tricky, they only
show up in the header section and not the body unless behavior was improved
in the last year or two. As I think I mentioned to you off-list somewhere
libcurl has minimal support for IMAP command parsing. You'll have to do a
lot of the parsing yourself. The actual command to get the subject for an
id is:
FETCH + id + BODY.PEEK[HEADER.FIELDS (SUBJECT)]
You may want to check out LibEtPan [2] or take a closer look at Vmime
which you've mentioned.
[1]: http://curl.haxx.se/libcurl/c/curl_easy_cleanup.html
[2]: https://github.com/dinhviethoa/libetpan
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
Interesting challenge. Can you just use different curl multi handles? --Daniel On Wed, May 27, 2015 at 1:10 PM, Michael Kaufmann m...@michael-kaufmann.ch wrote: Hi, I have a challenging scenario for libcurl. I want to connect with https to a cluster of ADFS hosts (Active Directory Federation Services). These hosts have the same hostname, but different IP addresses. For example: - host.example.org, IP: 10.0.0.1 - host.example.org, IP: 10.0.0.2 When establishing a connection, I want to control to which host libcurl should connect to (10.0.0.1 or 10.0.0.2) because the hosts have separate user sessions. Furthermore ADFS needs SNI, so it is necessary to use the hostname in the URL (https://host.example.org). It is not possible to just use the IP address (https://10.0.0.1/ or https://10.0.0.2/). As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: - CURLOPT_RESOLVE is not a local setting because it pre-populates the DNS cache. All easy handles that use the same multi handle share a DNS cache. Setting different IP addresses for the same hostname using CURLOPT_RESOLVE may therefore lead to race conditions. - A wrong existing connection may get reused. libcurl only looks at the hostname, and does not remember that the connection has been set up using a special IP address with CURLOPT_RESOLVE. (I have not tested this, but I have looked at libcurl's source code.) It is possible to work around these problems by using CURLOPT_RESOLVE with an artificial hostname in the URL, e.g. https://host.example.org-10.0.0.1/; - but then the server drops the connection, because the artificial hostname is also used for SNI. A libcurl option to control the SNI hostname (and also the hostname used for the server certificate checks) would make this workaround viable. A clean solution needs to address the problems mentioned above: - CURLOPT_RESOLVE should only affect a single easy handle - A new option to control the connection cache's behavior (reuse based on hostname only or reuse based on hostname + IP address). This may be implemented in a more general way with a label (string) that gets attached to a connection, and libcurl is only allowed to reuse connections when the labels match. What do you think? All ideas and suggestions are welcome. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html -- We must remember that those mortals we meet in parking lots, offices, elevators, and elsewhere are that portion of mankind God has given us to love and to serve. It will do us little good to speak of the general brotherhood of mankind if we cannot regard those who are all around us as our brothers and sisters.” --Spencer W. Kimball --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Question regarding multiple IMAP operations
On Wed, 27 May 2015, Adam wrote: Ray, I've actually written C89 code to parse IMAP responses. libcurl IMAP is indeed garbage, because the only thing it does for you that a straight socket doesn't is encryption ... and an non-blocking IMAP command/response state machine with a stable API that allows you to do IMAP communication with any number of IMAP servers in parallel... So as soon as I remove these crashes I will post my code to hopefully make it a bit more of a complete IMAP library. That'd be awesome! Let's say I want first list folders, the default, then 1 second later I want to EXAMINE OUTBOX, then 1 second later I want to read first message imap://imap.example.com/INBOX/;UID=1 There are no code samples showing how to do multiple requests like this. Re-use the handle, use the easy interface and just issue one request at a time. If you're fine with them being done in serial. Would something like this be the correct way? Or is this what the multi_ function families are for? multi is for non-blocking operations or several simultaneous ones. [code cut off] Looks perfectly reasonable from a quick glance! -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
On Wed, 27 May 2015, Michael Kaufmann wrote: As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: Let me suggeset yet another way with existing functionality: Use the share interface, and tell each easy handle that uses those host names to use its own share object with its own DNS cache. That way, the CURLOPT_RESOLVE setting would (should!) only affect that particular DNS cache. For two connections, some psuedo code would be: /* two share objects with DNS sharing */ share1 = curl_share_init() curl_share_setopt(share1, DNS); share2 = curl_share_init() curl_share_setopt(share2, DNS); curl_easy_setopt(easy1, CURLOPT_SHARE, share1); curl_easy_setopt(easy2, CURLOPT_RESOLVE, hostname:80:10.0.0.0); curl_easy_setopt(easy2, CURLOPT_SHARE, share2); curl_easy_setopt(easy2, CURLOPT_RESOLVE, hostname:80:192.168.0.1); ... then both handles would use the same host name but end up connecting to different IP addresses and yet use SNI and everything just fine. The drawback is of course that the handles now use one DNS cache each and don't benefit from the previous resolves from the other handles involved. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
Interesting challenge. Can you just use different curl multi handles? --Daniel Yes, I think this would work. But tracking more than one multi handle is a difficult task. libcurl is optimized for the add many easy handles to one multi handle approach. Regards, Michael On Wed, May 27, 2015 at 1:10 PM, Michael Kaufmann m...@michael-kaufmann.ch wrote: Hi, I have a challenging scenario for libcurl. I want to connect with https to a cluster of ADFS hosts (Active Directory Federation Services). These hosts have the same hostname, but different IP addresses. For example: - host.example.org, IP: 10.0.0.1 - host.example.org, IP: 10.0.0.2 When establishing a connection, I want to control to which host libcurl should connect to (10.0.0.1 or 10.0.0.2) because the hosts have separate user sessions. Furthermore ADFS needs SNI, so it is necessary to use the hostname in the URL (https://host.example.org). It is not possible to just use the IP address (https://10.0.0.1/ or https://10.0.0.2/). As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: - CURLOPT_RESOLVE is not a local setting because it pre-populates the DNS cache. All easy handles that use the same multi handle share a DNS cache. Setting different IP addresses for the same hostname using CURLOPT_RESOLVE may therefore lead to race conditions. - A wrong existing connection may get reused. libcurl only looks at the hostname, and does not remember that the connection has been set up using a special IP address with CURLOPT_RESOLVE. (I have not tested this, but I have looked at libcurl's source code.) It is possible to work around these problems by using CURLOPT_RESOLVE with an artificial hostname in the URL, e.g. https://host.example.org-10.0.0.1/; - but then the server drops the connection, because the artificial hostname is also used for SNI. A libcurl option to control the SNI hostname (and also the hostname used for the server certificate checks) would make this workaround viable. A clean solution needs to address the problems mentioned above: - CURLOPT_RESOLVE should only affect a single easy handle - A new option to control the connection cache's behavior (reuse based on hostname only or reuse based on hostname + IP address). This may be implemented in a more general way with a label (string) that gets attached to a connection, and libcurl is only allowed to reuse connections when the labels match. What do you think? All ideas and suggestions are welcome. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html -- We must remember that those mortals we meet in parking lots, offices, elevators, and elsewhere are that portion of mankind God has given us to love and to serve. It will do us little good to speak of the general brotherhood of mankind if we cannot regard those who are all around us as our brothers and sisters.” --Spencer W. Kimball --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
On Wed, 27 May 2015, Michael Kaufmann wrote: Can you just use different curl multi handles? Yes, I think this would work. But tracking more than one multi handle is a difficult task. libcurl is optimized for the add many easy handles to one multi handle approach. Yes it is. We've been in this area before though, and maybe we should at some point consider a way to work better with multiple multi handles... -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Happy eyeballs and custom DNS
Hi We're using our own DNS manager and integrate with curl with CURLOPT_RESOLVE. Will Curl do happy eyeballs if both an ipv4 and an ipv6 result is provided? regards -- Anders Bakken - Netflix --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Happy eyeballs and custom DNS
On Wed, 27 May 2015, Anders Bakken via curl-library wrote: We're using our own DNS manager and integrate with curl with CURLOPT_RESOLVE. Will Curl do happy eyeballs if both an ipv4 and an ipv6 result is provided? No, unfortunately not. It was discussed briefly not long ago, but CURLOPT_RESOLVE still only offers a single address for the given name+port combo. It would be really neat to have it support multiple addresses. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Question regarding multiple IMAP operations
On 5/26/2015 9:31 AM, Adam wrote: What is the proper way to send multiple IMAP commands? For example, first I'd like to get a listing of all the folder names, then I want to list the unread files in the folder names. I've tried using the same CURL* curl handle, sometimes it crashes on curl_easy_cleanup() (single threaded app), and sometimes it crashes on the second call to curl_easy_perform(). It depends on the IMAP server, gmail causes a crash, hotmail doesn't. I've even tried opening/closing a new handle for each operation and this causes crashes also. What is the proper way to do this? There are no code samples for this on the libcurl homepage. Thanks. The proper way is multiple requests and don't call curl_easy_cleanup until after you're done with the handle. You want to reuse the handle until then. Please note as mentioned in the curl_easy_cleanup doc that Any use of the handle after this function has been called and have returned, is illegal. [1] So you have custom request(s) to get the unread ids then you get the subjects. Getting the subjects from what I remember is tricky, they only show up in the header section and not the body unless behavior was improved in the last year or two. As I think I mentioned to you off-list somewhere libcurl has minimal support for IMAP command parsing. You'll have to do a lot of the parsing yourself. The actual command to get the subject for an id is: FETCH + id + BODY.PEEK[HEADER.FIELDS (SUBJECT)] You may want to check out LibEtPan [2] or take a closer look at Vmime which you've mentioned. [1]: http://curl.haxx.se/libcurl/c/curl_easy_cleanup.html [2]: https://github.com/dinhviethoa/libetpan --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Question regarding multiple IMAP operations
On 5/27/2015 3:09 PM, Adam wrote:
Ray, I've actually written C89 code to parse IMAP responses. libcurl
IMAP is indeed garbage, because the only thing it does for you that a
straight socket doesn't is encryption (which is well documented w/tons
of code samples anyway). So as soon as I remove these crashes I will
post my code to hopefully make it a bit more of a complete IMAP
library. I've already come this far w/libcurl despite it's awkward
callback schemes so don't feel like switching now.
The imap code isn't garbage it's just limited in what it can do. That's
very different from bad code. If you have contributions to make to
imap.c that's great. Contributions aren't guaranteed to be added,
however. If the changes you are making are substantial you should
discuss it to get some feedback and a better idea if what you are
writing is something that could be accepted. That way you won't waste
time. And please review the coding guidelines at [1].
I am not the best person to do a review for whatever you propose so you
will probably not hear much from me on that. My IMAP experience is
limited to using regular expressions to parse results from the servers
and is not in any way RFC compliant.
Let's say I want first list folders, the default, then 1 second later
I want to EXAMINE OUTBOX, then 1 second later I want to read first
message imap://imap.example.com/INBOX/;UID=1
http://imap.example.com/INBOX/;UID=1 There are no code samples
showing how to do multiple requests like this.
Would something like this be the correct way? Or is this what the
multi_ function families are for? Not this exactly, but doing stuff
like this I'm getting a lot of different crashes, and I'm not using
any handles after calling cleanup.
The code is incorrect you want to keep the same scheme host port in the
url so the connection is reused. You want to use CURLOPT_CUSTOMREQUEST
instead of CURLOPT_URL to send a custom request. So for example
curl_easy_setopt(curl, CURLOPT_URL, imaps://imap.gmail.com:993/);
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, EXAMINE INBOX);
you should see in the body the results of the examine command. if you
want to then fetch INBOX email id 1 you'd do:
curl_easy_setopt(curl, CURLOPT_URL,
imaps://imap.gmail.com:993/INBOX/;UID=1);
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, NULL);
now that e-mail is in the body in addition to examine results (assuming
you didn't clear the memory in between).
The thing about IMAP support is certain commands have limited support
depending on how you format the URL (APPEND, SELECT, FETCH, LIST,
SEARCH) and other commands you have to use CUSTOMREQUEST. The latter is
considered exactly that, a custom request, so the parsing is not done
for you. In those cases you will find the results written to the header,
not the body. I encourage you to review imap.c [3].
Since you have already written IMAP parsing you may already be aware of
this but when you request a URL like this:
imaps://imap.example.com/INBOX/;UID=1
what actually happens is these two commands
SELECT INBOX
FETCH 1 BODY[]
That behavior is specified according to the RFC [2]. Since libcurl sent
the command it recognizes the response, so it should parse it out and
put it in the body. If you send FETCH 1 BODY[] as a custom command
though the response is not parsed out so it's in the header section
instead of the body.
And as an example of limitation libcurl can't handle %0D%0A in the URL
even though it's technically allowed in the urls according to the RFC.
That maybe? could be a problem during search but if you don't use the {}
for data block, you can still do it straightaway like
imaps://imap.gmail.com:993/INBOX?charset%20UTF-8%20SUBJECT%20test
which actually does:
SELECT INBOX
SEARCH charset UTF-8 SUBJECT test
and your body will contain a response of
* SEARCH 1 2 3 4
or whatever
Also the etiquette for this mailing list is no top-posting [4].
[1]: http://curl.haxx.se/dev/contribute.html#Naming
[2]: https://tools.ietf.org/html/rfc5092
[3]: https://github.com/bagder/curl/blob/master/lib/imap.c
[4]: http://curl.haxx.se/mail/etiquette.html#Do_Not_Top_Post
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Happy eyeballs and custom DNS
How much work do you think it would be? (and could you point me in the general right direction for it?) Anders On Wed, May 27, 2015 at 2:39 PM, Daniel Stenberg dan...@haxx.se wrote: On Wed, 27 May 2015, Anders Bakken via curl-library wrote: We're using our own DNS manager and integrate with curl with CURLOPT_RESOLVE. Will Curl do happy eyeballs if both an ipv4 and an ipv6 result is provided? No, unfortunately not. It was discussed briefly not long ago, but CURLOPT_RESOLVE still only offers a single address for the given name+port combo. It would be really neat to have it support multiple addresses. -- / daniel.haxx.se -- Anders Bakken - Netflix --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Happy eyeballs and custom DNS
On 5/27/2015 6:51 PM, Anders Bakken via curl-library wrote: How much work do you think it would be? (and could you point me in the general right direction for it?) I worked on it a while ago. Stefan had some changes to add address remove capability that I thought pre-empted mine so his were added instead with the plan that mine would make it in at some later date. I have to adjust my code to mesh with those changes. It's a low priority for me, it's been on the backburner. I never tested happy eyeballs with my changes. http://curl.haxx.se/mail/lib-2015-03/0012.html https://github.com/bagder/curl/compare/master...jay:fix-curlopt_resolve?expand=1 --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
