Re: Phantom socket with threaded resolver

[Sean Miller via curl-library]

> You can't and you shouldn't. libcurl doesn't tell the application when the 
> DNS 
> resolution is done (or even if DNS resolution is at all performed). You can 
> still attempt to detect it, but there's no promise it will work.

Ack, thanks.

For future reference, my solution was to do the legwork to specially handle the 
AF_UNIX socketpair() socket.

Regards,
Sean M.

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Phantom socket with threaded resolver

[Sean Miller via curl-library]

> (This might be a varation of this issue:
> https://github.com/curl/curl/issues/5747 )

Yes, I think you're correct. I looked for an existing issue but did not find 
this one.

> That's correct, because it isn't a regular socket. It's a socketpair. 
> CURLMOPT_SOCKETFUNCTION should get told about it though. Doesn't it?

Yes, libcurl tells me about the socketpair. It's apparently AF_INET domain, too.

Maybe I'm dense, but how can I tell that I should treat this invocation of
CURLMOPT_SOCKETFUNCTION as a signal that DNS resolution is done? I'll
think more about this.

Thanks for your help. You're immensely responsive.

Regards,
Sean M.
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Phantom socket with threaded resolver

[Sean Miller via curl-library]

Hello again, libcurl experts. I'm using libcurl's threaded resolver. A trimmed 
log excerpt follows:

19:39:19,837 DEBUG [libcurl_multi_wrapper] - adding easy handle
19:39:19,837 DEBUG [http_client] - on_libcurl_multi_timer(timeout_ms=0)
19:39:19,837 DEBUG [http_client] - on_timeout
19:39:19,837 DEBUG [libcurl_multi_wrapper] - performing socket action; fd=-1, 
event_bitmask=0
19:39:19,838 DEBUG [http_connection] - from libcurl: STATE: INIT => CONNECT 
handle 0x7f6d7ad8; line 1491 (connection #-5000)
19:39:19,838 DEBUG [http_connection] - from libcurl: Added connection 0. The 
cache now contains 1 members
19:39:19,838 DEBUG [http_connection] - from libcurl: STATE: CONNECT => 
WAITRESOLVE handle 0x7f6d7ad8; line 1532 (connection #0)
19:39:19,838 DEBUG [http_client] - on_libcurl_multi_socket(sock=11, what=IN)
19:39:19,838 DEBUG [libcurl_socket_manager] - watch(sock=11, action=IN)
19:39:19,838 WARN  [libcurl_socket_manager] - asked to watch unmanaged socket

Here, I add an easy handle to a multi handle and call 
curl_multi_socket_action() with CURL_SOCKET_TIMEOUT and event 0 to kick off the 
transaction.

In response, libcurl appears to create a socket for DNS resolution without 
asking for it via CURLOPT_OPENSOCKETFUNCTION. Unfortunately, my side of the 
program isn't aware of that socket and so cannot watch it for read-readiness as 
requested. Subsequently, libcurl requests a series of timeouts of increasing 
duration, which typically delay the first transactions by 64-256 ms.

Compiling libcurl with --disable-threaded-resolver removes the WAITRESOLVE 
state, and, therefore, the phantom socket.

Is this behavior intentional? Is there a recommended way to either force 
libcurl to ask for the socket, or else avoid telling me about it? Where did I 
go wrong?

Regards,
Sean M.

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Static linking with ssl

[Sean Miller via curl-library]

> I get lots of linking errors (missing symbols) – not when compiling curl, but 
> when I try to statically link libcurl to my application. The missing symbols 
> are all SSL related (I can dig them out if is of any use).



Statically linking doesn't include libcurl's own dependencies.
Find out which libraries provide the missing symbols and add them to your link 
flags.
Regards,Sean M.   ---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html

Re: Incomplete HTTP post body

[Sean Miller via curl-library]

On Friday, August 5, 2016 12:49 AM, Daniel Stenberg  wrote:



> Please don't top-post.

I can't remember the last time I used e-mail like this. Two decades, maybe. 
It's cool. Have to do it manually, though.

>> For a large request (>64 KiB) with a timeout of 5 sec, I see libcurl 
>> requesting up to 4 buffers worth of data from my READFUNC, with one call 
>> to TIMERFUNC with timeout_ms=198. After the 4th chunk, libcurl calls 
>> TIMERFUNC with timeout_ms=4800, which consumes all the remaining time until 
>> the timeout. I don't understand why libcurl wants to wait so long when I 
>> haven't told it that I'm done providing data.

> libcurl will wait for the socket to become writable until it asks for data 
> from you, so that it knows it can send at least parts of the data you give 
> it. 
> libcurl avoids to buffer data by itself. So, until the socket signals 
> "writable", your read callback won't get called. And then it won't ask for 
> more data from you again until it has managed to send the data you gave it in 
> the previous callback call.

This led me to the solution. The asiohiper example misled me into believing 
libcurl would wait for me, but it doesn't. Unlike libev and libevent, asio 
doesn't install wait loops; the user must loop manually after each wait. Thus, 
I wrote some async loops over the io_service, and I was able to tell libcurl 
when to write again.
Once I have some free time, I'd like to fix your example.
Regards,Sean M.  ---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html

Re: Incomplete HTTP post body

[Sean Miller via curl-library]

Thanks for your advice.
I cannot control which version of libcurl is deployed on these hosts. I can, 
however, statically compile curl-7_50_1. So I did.
For a large request (>64 KiB) with a timeout of 5 sec, I see libcurl requesting 
up to 4 buffers worth of data from my READFUNC, with one call to TIMERFUNC with 
timeout_ms=198. After the 4th chunk, libcurl calls TIMERFUNC with 
timeout_ms=4800, which consumes all the remaining time until the timeout. I 
don't understand why libcurl wants to wait so long when I haven't told it that 
I'm done providing data.
If you can intuit from this what my mistake might have been, I'd immensely 
appreciate that.
In the meantime, I'm going to try to pare down my code to a minimal 
reproducible example...
Regards,Sean M. 

On Thursday, August 4, 2016 12:24 AM, Daniel Stenberg <dan...@haxx.se> 
wrote:
 

 On Wed, 3 Aug 2016, Sean Miller via curl-library wrote:

> I'm using libcurl-7.19.7 to post JSON to an HTTP server. I'm aware this is 
> an old version. I'm using the multi interface, and I based my implementation 
> on the example given in asiohiper.cpp. I worked around bug #62.

We fixed known bug #62 in 7.21.2 with a commit merged nearly 6 years ago...

> Problem: I observe incomplete transmission of request bodies larger than 
> about 40K bytes. Smaller requests are successful.

I can't recall any particular bug that would show up like that. I supposes it 
would depend on exactly what more happens on that transation. Chances are also 
that I've just forgotten about some old flaw.

But why bother? You're probably suffering from a bug in libcurl so if you want 
that fixed you need to patch your libcurl and rebuild, and if you can do that 
surely you're better off just considering upgrading instead and save yourself 
a lot of troubles and quite likely fix a few other bugs at the same time that 
are waiting to strike down on you soon...

If you're looking for the minimal patch to fix only this problem, then I'd 
recommend "git bisect" to find the fix that made the problem go away and then 
you backport that one to your version.

-- 

  / daniel.haxx.se


  ---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html

Incomplete HTTP post body

[Sean Miller via curl-library]

I'm using libcurl-7.19.7 to post JSON to an HTTP server. I'm aware this is an 
old version. I'm using the multi interface, and I based my implementation on 
the example given in asiohiper.cpp. I worked around bug #62.
Problem: I observe incomplete transmission of request bodies larger than about 
40K bytes. Smaller requests are successful.
I set Content-{Type,Length} myself. I have tried the following two ways of 
supplying the body to libcurl:  1) CURLOPT_POSTFIELDS with 
CURLOPT_POSTFIELDSIZE, and  2) CURLOPT_READFUNCTION with CURLOPT_READDATA.
In case #1, tcpdump shows that libcurl only sends partial data. The remote 
server acks each packet sent.
In case #2, my logs show that libcurl only invokes my read callback for some of 
the data, usually no more than two 16 KiB buffers worth.
In both cases, the transfer eventually times out because the remote server does 
not respond to the incomplete post.
Any pointers? Thanks in advance.
Regards,Sean---
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html