Re: Phantom socket with threaded resolver

[Sean Miller via curl-library]

> You can't and you shouldn't. libcurl doesn't tell the application when the 
> DNS 
> resolution is done (or even if DNS resolution is at all performed). You can 
> still attempt to detect it, but there's no promise it will work.

Ack, thanks.

For future reference, my solution was to do the legwork to specially handle the 
AF_UNIX socketpair() socket.

Regards,
Sean M.

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Phantom socket with threaded resolver

[Sean Miller via curl-library]

> (This might be a varation of this issue:
> https://github.com/curl/curl/issues/5747 )

Yes, I think you're correct. I looked for an existing issue but did not find 
this one.

> That's correct, because it isn't a regular socket. It's a socketpair. 
> CURLMOPT_SOCKETFUNCTION should get told about it though. Doesn't it?

Yes, libcurl tells me about the socketpair. It's apparently AF_INET domain, too.

Maybe I'm dense, but how can I tell that I should treat this invocation of
CURLMOPT_SOCKETFUNCTION as a signal that DNS resolution is done? I'll
think more about this.

Thanks for your help. You're immensely responsive.

Regards,
Sean M.
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Phantom socket with threaded resolver

[Sean Miller via curl-library]

Hello again, libcurl experts. I'm using libcurl's threaded resolver. A trimmed 
log excerpt follows:

19:39:19,837 DEBUG [libcurl_multi_wrapper] - adding easy handle
19:39:19,837 DEBUG [http_client] - on_libcurl_multi_timer(timeout_ms=0)
19:39:19,837 DEBUG [http_client] - on_timeout
19:39:19,837 DEBUG [libcurl_multi_wrapper] - performing socket action; fd=-1, 
event_bitmask=0
19:39:19,838 DEBUG [http_connection] - from libcurl: STATE: INIT => CONNECT 
handle 0x7f6d7ad8; line 1491 (connection #-5000)
19:39:19,838 DEBUG [http_connection] - from libcurl: Added connection 0. The 
cache now contains 1 members
19:39:19,838 DEBUG [http_connection] - from libcurl: STATE: CONNECT => 
WAITRESOLVE handle 0x7f6d7ad8; line 1532 (connection #0)
19:39:19,838 DEBUG [http_client] - on_libcurl_multi_socket(sock=11, what=IN)
19:39:19,838 DEBUG [libcurl_socket_manager] - watch(sock=11, action=IN)
19:39:19,838 WARN  [libcurl_socket_manager] - asked to watch unmanaged socket

Here, I add an easy handle to a multi handle and call 
curl_multi_socket_action() with CURL_SOCKET_TIMEOUT and event 0 to kick off the 
transaction.

In response, libcurl appears to create a socket for DNS resolution without 
asking for it via CURLOPT_OPENSOCKETFUNCTION. Unfortunately, my side of the 
program isn't aware of that socket and so cannot watch it for read-readiness as 
requested. Subsequently, libcurl requests a series of timeouts of increasing 
duration, which typically delay the first transactions by 64-256 ms.

Compiling libcurl with --disable-threaded-resolver removes the WAITRESOLVE 
state, and, therefore, the phantom socket.

Is this behavior intentional? Is there a recommended way to either force 
libcurl to ask for the socket, or else avoid telling me about it? Where did I 
go wrong?

Regards,
Sean M.

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Asiohiper example program delay

[Sean Miller]

> I think you've stepped on a libcurl bug!

Uh oh. I'll have to clean off my shoe.

> Are you doing HTTPS through a HTTP proxy by any chance? That's what the 
> PROTOCONNECT state is for basically.

Yes, almost everything I do in this large private cloud goes through proxies.

> Does it seem like that could be what happens in your case? If so, I'll write 
> up a patch you can test...

I'd be happy to test a patch. I've been poking around in the code a little 
myself, but I'm unfamiliar with it.

I've reproduced this on libcurl-7.19.7 (RHEL6... I know...) and libcurl-7.55.1, 
the latter of which I build myself.

Regards,
Sean M.
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Asiohiper example program delay

[Sean Miller]

I added simple logging and a debug function to the following Boost ASIO-based 
example program:

https://curl.haxx.se/libcurl/c/asiohiper.html

Summary:

1) In the WAITCONNECT state, libcurl calls multi_timer_cb(200) and registers 
interest in writing.
2) The FSM transitions first to SENDPROTOCONNECT and then to PROTOCONNECT.
3) Libcurl calls remsock() through sock_cb().
4) ... Nothing happens until the timer fires ~165 ms later.

What should the example do, if possible, to avoid this delay? I appreciate any 
ideas/clues/hints. Until then, I'm investigating the differences between this 
and the other example programs.

I'm including the log (trimmed to point of interest) below:

2018-05-17 21:06:11,461 INFO  root - opensocket:
2018-05-17 21:06:11,461 INFO  root - from libcurl:   Trying 172.217.12.4...
2018-05-17 21:06:11,461 INFO  root - from libcurl: TCP_NODELAY set
2018-05-17 21:06:11,461 INFO  root - from libcurl: STATE: WAITRESOLVE => 
WAITCONNECT handle 0x1d97188; line 1539 (connection #0)
2018-05-17 21:06:11,461 INFO  root - sock_cb (sock=6, what=2)
2018-05-17 21:06:11,461 INFO  root - sock_cb: adding data: OUT
2018-05-17 21:06:11,461 INFO  root - addsock (action=2)
2018-05-17 21:06:11,461 INFO  root - setsock (sock=6, action=2, oldact=0)
2018-05-17 21:06:11,461 INFO  root - setsock: watching for socket to become 
writable
2018-05-17 21:06:11,461 INFO  root - multi_timer_cb (timeout_ms=200)
2018-05-17 21:06:11,461 INFO  root - check_multi_info; remaining=1
2018-05-17 21:06:11,494 INFO  root - event_cb (sock=6, action=2)
2018-05-17 21:06:11,494 INFO  root - event_cb: running multi socket action
2018-05-17 21:06:11,494 INFO  root - from libcurl: Connected to www.google.com 
(172.217.12.4) port 80 (#0)
2018-05-17 21:06:11,494 INFO  root - from libcurl: STATE: WAITCONNECT => 
SENDPROTOCONNECT handle 0x1d97188; line 1591 (connection #0)
2018-05-17 21:06:11,494 INFO  root - from libcurl: Marked for [keep alive]: 
HTTP default
2018-05-17 21:06:11,494 INFO  root - from libcurl: STATE: SENDPROTOCONNECT => 
PROTOCONNECT handle 0x1d97188; line 1605 (connection #0)
2018-05-17 21:06:11,494 INFO  root - sock_cb (sock=6, what=4)
2018-05-17 21:06:11,494 INFO  root - remsock
2018-05-17 21:06:11,494 INFO  root - check_multi_info; remaining=1
2018-05-17 21:06:11,494 INFO  root - event_cb: keep watching...?
2018-05-17 21:06:11,662 INFO  root - timer_cb: 
2018-05-17 21:06:11,662 INFO  root - from libcurl: STATE: PROTOCONNECT => DO 
handle 0x1d97188; line 1626 (connection #0)
2018-05-17 21:06:11,662 INFO  root - from libcurl: STATE: DO => DO_DONE handle 
0x1d97188; line 1688 (connection #0)
2018-05-17 21:06:11,662 INFO  root - from libcurl: STATE: DO_DONE => 
WAITPERFORM handle 0x1d97188; line 1813 (connection #0)
2018-05-17 21:06:11,662 INFO  root - from libcurl: STATE: WAITPERFORM => 
PERFORM handle 0x1d97188; line 1823 (connection #0)
2018-05-17 21:06:11,662 INFO  root - sock_cb (sock=6, what=1)
2018-05-17 21:06:11,662 INFO  root - sock_cb: adding data: IN
2018-05-17 21:06:11,662 INFO  root - addsock (action=1)
2018-05-17 21:06:11,662 INFO  root - setsock (sock=6, action=1, oldact=0)
2018-05-17 21:06:11,662 INFO  root - setsock: watching for socket to become 
readable
2018-05-17 21:06:11,662 INFO  root - multi_timer_cb (timeout_ms=1000)
2018-05-17 21:06:11,662 INFO  root - check_multi_info; remaining=1

Thanks,
Sean M.



---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Static linking with ssl

[Sean Miller via curl-library]

> I get lots of linking errors (missing symbols) – not when compiling curl, but 
> when I try to statically link libcurl to my application. The missing symbols 
> are all SSL related (I can dig them out if is of any use).



Statically linking doesn't include libcurl's own dependencies.
Find out which libraries provide the missing symbols and add them to your link 
flags.
Regards,Sean M.   ---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html

Re: Incomplete HTTP post body

[Sean Miller via curl-library]

On Friday, August 5, 2016 12:49 AM, Daniel Stenberg  wrote:



> Please don't top-post.

I can't remember the last time I used e-mail like this. Two decades, maybe. 
It's cool. Have to do it manually, though.

>> For a large request (>64 KiB) with a timeout of 5 sec, I see libcurl 
>> requesting up to 4 buffers worth of data from my READFUNC, with one call 
>> to TIMERFUNC with timeout_ms=198. After the 4th chunk, libcurl calls 
>> TIMERFUNC with timeout_ms=4800, which consumes all the remaining time until 
>> the timeout. I don't understand why libcurl wants to wait so long when I 
>> haven't told it that I'm done providing data.

> libcurl will wait for the socket to become writable until it asks for data 
> from you, so that it knows it can send at least parts of the data you give 
> it. 
> libcurl avoids to buffer data by itself. So, until the socket signals 
> "writable", your read callback won't get called. And then it won't ask for 
> more data from you again until it has managed to send the data you gave it in 
> the previous callback call.

This led me to the solution. The asiohiper example misled me into believing 
libcurl would wait for me, but it doesn't. Unlike libev and libevent, asio 
doesn't install wait loops; the user must loop manually after each wait. Thus, 
I wrote some async loops over the io_service, and I was able to tell libcurl 
when to write again.
Once I have some free time, I'd like to fix your example.
Regards,Sean M.  ---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html

Re: Incomplete HTTP post body

[Sean Miller via curl-library]

Thanks for your advice.
I cannot control which version of libcurl is deployed on these hosts. I can, 
however, statically compile curl-7_50_1. So I did.
For a large request (>64 KiB) with a timeout of 5 sec, I see libcurl requesting 
up to 4 buffers worth of data from my READFUNC, with one call to TIMERFUNC with 
timeout_ms=198. After the 4th chunk, libcurl calls TIMERFUNC with 
timeout_ms=4800, which consumes all the remaining time until the timeout. I 
don't understand why libcurl wants to wait so long when I haven't told it that 
I'm done providing data.
If you can intuit from this what my mistake might have been, I'd immensely 
appreciate that.
In the meantime, I'm going to try to pare down my code to a minimal 
reproducible example...
Regards,Sean M. 

On Thursday, August 4, 2016 12:24 AM, Daniel Stenberg <dan...@haxx.se> 
wrote:
 

 On Wed, 3 Aug 2016, Sean Miller via curl-library wrote:

> I'm using libcurl-7.19.7 to post JSON to an HTTP server. I'm aware this is 
> an old version. I'm using the multi interface, and I based my implementation 
> on the example given in asiohiper.cpp. I worked around bug #62.

We fixed known bug #62 in 7.21.2 with a commit merged nearly 6 years ago...

> Problem: I observe incomplete transmission of request bodies larger than 
> about 40K bytes. Smaller requests are successful.

I can't recall any particular bug that would show up like that. I supposes it 
would depend on exactly what more happens on that transation. Chances are also 
that I've just forgotten about some old flaw.

But why bother? You're probably suffering from a bug in libcurl so if you want 
that fixed you need to patch your libcurl and rebuild, and if you can do that 
surely you're better off just considering upgrading instead and save yourself 
a lot of troubles and quite likely fix a few other bugs at the same time that 
are waiting to strike down on you soon...

If you're looking for the minimal patch to fix only this problem, then I'd 
recommend "git bisect" to find the fix that made the problem go away and then 
you backport that one to your version.

-- 

  / daniel.haxx.se


  ---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html

Incomplete HTTP post body

[Sean Miller via curl-library]

I'm using libcurl-7.19.7 to post JSON to an HTTP server. I'm aware this is an 
old version. I'm using the multi interface, and I based my implementation on 
the example given in asiohiper.cpp. I worked around bug #62.
Problem: I observe incomplete transmission of request bodies larger than about 
40K bytes. Smaller requests are successful.
I set Content-{Type,Length} myself. I have tried the following two ways of 
supplying the body to libcurl:  1) CURLOPT_POSTFIELDS with 
CURLOPT_POSTFIELDSIZE, and  2) CURLOPT_READFUNCTION with CURLOPT_READDATA.
In case #1, tcpdump shows that libcurl only sends partial data. The remote 
server acks each packet sent.
In case #2, my logs show that libcurl only invokes my read callback for some of 
the data, usually no more than two 16 KiB buffers worth.
In both cases, the transfer eventually times out because the remote server does 
not respond to the incomplete post.
Any pointers? Thanks in advance.
Regards,Sean---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html