Re: should curl_multi_timeout() account for CURLOPT_LOW_SPEED_TIME?

[Daniel Stenberg]

On Mon, 10 Apr 2017, Rainer Canavan wrote:

A patch that turns docs/examples/multi-app.c into a reproducer, as well as 
the trivial PHP program used as a source is attached. When i run it ( > 
/dev/null) it shows that curl_multi_timeout returns values >> 1000ms:


Thanks a lot for that. Using that code it was easy to reproduce your issue!

I propose the fix I've submitted as PR #1407 [1]. It would be lovely if you 
could try it out and see if it fixes the symtoms for you as well!


[1] = https://github.com/curl/curl/pull/1407

--

 / daniel.haxx.se
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: FTPS: "SSL certificate problem: Invalid certificate chain" error

[TJ Saunders]

> This is not a rule of thumb, but most FTP servers tell you whether they 
> support EPSV upon receiving the "HELP" command. The HELP reply format is 
> unfortunately not structured and may differ from one server to another.

Alternatively, you could use the FEAT command.  Most FTP servers which
support EPSV will list EPSV (and/or EPRT) in their FEAT response.  The
FEAT response is much more structured, making it easier to parse.

See:

  https://tools.ietf.org/html/rfc2389#section-3.2

Hope this helps,
TJ
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: should curl_multi_timeout() account for CURLOPT_LOW_SPEED_TIME?

[Daniel Stenberg via curl-library]

On Mon, 10 Apr 2017, Rainer Canavan via curl-library wrote:

I'm not entirely sure why that is so, but I'd like to mention that the 
CURLOPT_LOW_SPEED_TIME handling and its timeouts were just now changed, in 
commit 29147ade0456a which landed just hours ago.


I can't find that commit anywhere, is that 2d5711dc11 in the github 
repository?


Wow, I don't know how I got that weird hash (probably from a different/wrong 
branch) but yes I meant 2d5711dc11.


When CURLOPT_LOW_SPEED_TIME is set, libcurl should now use no longer than 
1000ms timeouts.


I've just merged 2d5711dc11 into 7.53.1, and I don't really see an 
improvement.


Then we clearly need to debug this more. Can you share code with us that 
reproduces this behavior easily ?


The other thing is that I would have expected CURLOPT_LOW_SPEED_TIME to be 
the actual interval over which the speed is measured, i.e. if a server does 
not send any data for 2 * CURLOPT_LOW_SPEED_TIME seconds, the transfer 
should always fail. The actual implementation uses progress.current_speed, 
which is averaged over longer periods with the result that bursty transfers 
are kept going.


It is an average speed over the last 5 seconds, yes (or less if the transfer 
is younger). Transfer speeds with its bursts and what not is not that easy to 
get "right" by everyone's view, but I think that this is a decent approach.


Well, that is *after* this recent fix because previously it was actually even 
worse! =)


That's perfectly fine, and probably more useful in reality than how I 
thought it would work, but I would say the documentation could be clearer.


Feel free to suggest improvements to the documentation!

--

 / daniel.haxx.se
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: should curl_multi_timeout() account for CURLOPT_LOW_SPEED_TIME?

[Rainer Canavan via curl-library]

On Thu, Apr 6, 2017 at 9:04 PM, Daniel Stenberg  wrote:
> On Thu, 6 Apr 2017, Rainer Canavan wrote:
>
>> it looks like curl_multi_timeout() doesn't always respect
>> CURLOPT_LOW_SPEED_TIME.
>
> I'm not entirely sure why that is so, but I'd like to mention that the
> CURLOPT_LOW_SPEED_TIME handling and its timeouts were just now changed, in
> commit 29147ade0456a which landed just hours ago.

I can't find that commit anywhere, is that 2d5711dc11 in the github repository?

> When CURLOPT_LOW_SPEED_TIME is set, libcurl should now use no longer than
> 1000ms timeouts.

I've just merged 2d5711dc11 into 7.53.1, and I don't really see an improvement.
curl_multi_timeout() still returns the same erratic values, but
docs/examples/multi-app.c
protects itself by limiting the select timeout to 1s, so I suppose I'll have
to do the same.

The other thing is that I would have expected CURLOPT_LOW_SPEED_TIME to
be the actual interval over which the speed is measured, i.e. if a
server does not
send any data for 2 * CURLOPT_LOW_SPEED_TIME seconds, the transfer should
always fail. The actual implementation uses progress.current_speed, which is
averaged over longer periods with the result that bursty transfers are
kept going.
That's perfectly fine, and probably more useful in reality than how I thought it
would work, but I would say the documentation could be clearer.


Rainer
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: FTPS: "SSL certificate problem: Invalid certificate chain" error

[Patrick Monnerat]

On 04/10/2017 08:07 AM, Leo wrote:


This code is in a tool that's been working for years by now. I 
especially disabled EPSV because it caused connection issues with many 
FTP servers. With EPSV off, there were no problems so far.



Because so far, you were'nt in a NAT+SSL+PASV condition...

But support for FTPS is relatively recent.

Not really; libcurl supports it for more than 8 years, AFAICR !


So I wonder if you could shed some light on this - or point me to 
sources... Is there a way to determine that server requires EPSV 
instead of PASV?
libcurl tries EPSV first (if not disabled); if it fails, it then reverts 
to PASV, thus it is generally safe to leave EPSV enabled.


A server itself will never require EPSV: it might support it or not. 
What makes it required is the NAT+SSL condition.


This is not a rule of thumb, but most FTP servers tell you whether they 
support EPSV upon receiving the "HELP" command. The HELP reply format is 
unfortunately not structured and may differ from one server to another.


The difference is: PASV reply tells the client the port number and the 
IP address to use for a data connection, while EPSV only tells the port 
number and the client should use the same IP address as for the control 
connection.


For FTP protocol details, you can have a look at 
https://www.rfc-editor.org/info/rfc959 and for the EPSV extension, 
https://www.rfc-editor.org/info/rfc2428.


Code issuing the PASV or EPSV command in libcurl is in 
https://github.com/curl/curl/blob/master/lib/ftp.c function 
ftp_state_use_pasv(), while EPSV failure is handled in function 
ftp_state_pasv_resp().


Patrick
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: FTPS: "SSL certificate problem: Invalid certificate chain" error

[Leo]

On 4/9/17 5:25 AM, Patrick Monnerat wrote:
You probably have disabled EPSV: obviously your server is behind a 
NATting firewall: its public address is 125.142.62.86, but it requires 
you to connect to its private address (192...). There is a well known 
problem with ftp when using PASV + SSL + NAT:
- The transmitted data connection address is private, thus unreachable 
from outside the server's net,
- NATting device deep inspection cannot translate it to the 
corresponding public address it because the connection is encrypted.


The only way to overcome this caveat is to suppress the use of one of 
the PASV/SSL/NAT feature. If the target server supports it, the easier 
would be to use EPSV instead of PASV.


You probably have a code line in your program such as:
  curl_easy_setopt(handle, CURLOPT_FTP_USE_EPSV, 0L);

Remove it and the trick should be done :-)


Thanks for the detailed info, Patrick!

That worked indeed!

Now I wonder if there's a way to recognize that a server requires EPSV 
before connecting?


This code is in a tool that's been working for years by now. I 
especially disabled EPSV because it caused connection issues with many 
FTP servers. With EPSV off, there were no problems so far.


But support for FTPS is relatively recent.

So I wonder if you could shed some light on this - or point me to 
sources... Is there a way to determine that server requires EPSV instead 
of PASV?



Thanks,
Leo

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html