On Tuesday, 17 April 2012 at 07:07:06 UTC, Gleb wrote:
Gentlemen,
While working with curl library, I would like to discuss a few
more issues I faced with, if you don't mind.
Here is the code sample I use:
auto f = new std.stream.BufferedFile("file.zip",
FileMode.In);
scope (exit) f.close();
auto client = FTP();
client.verbose(true);
client.url = "ftp://192.168.110.58/file.zip";;
client.setAuthentication("user", "pass");
client.contentLength = cast(size_t)f.size;
client.handle.set(CurlOption.upload, 1L);
client.onSend = (void[] data)
{
return f.read(cast(ubyte[])data);
};
client.perform();
1. TIMEOUTS
The code sample will work great if file.zip is small.
If the file is big enough, we will get the following result:
* Connecting to 192.168.110.58 (192.168.110.58) port 1468
> TYPE I
< 200 Type set to I
> STOR file.zip
< 150 Opening BINARY mode data connection for file.zip
* Operation timed out after 120105 milliseconds with
687751168 bytes received
* Closing connection #0
* Timeout was reached
std.net.curl.CurlTimeoutException@std\net\curl.d(3348): Timeout
was reached on handle 16BCB60
So we were disconnected from server because of the timeout. To
avoid this behavior I use the following:
client.dataTimeout(dur!"weeks"(10));
I'm not sure if we really need this kind of timeout so I use
the really big value. Anyway, I believe the default value of
120 seconds is not enough for practical usage of file transfer
protocol. Maybe it's suitable for HTTP only?
By the way, for some reason I can't use the value more then 10
weeks. In this case we can't even connect to server:
* About to connect() to 192.168.110.58 port 21 (#0)
* Trying 192.168.110.58...
* connected
* Connected to 192.168.110.58 (192.168.110.58) port 21 (#0)
* server response timeout
* Closing connection #0
* Timeout was reached
std.net.curl.CurlTimeoutException@std\net\curl.d(3348):
Timeout was reached on handle 164CB60
Of course, even bigger value is not necessary, but I'm not sure
this behavior is correct.
This is one of the many reasons I believe we should do our own
network library. Curl only support timeouts for connecting and
for the entire transfer. What you really want is better control
with sane defaults: DNS lookup timeout, connect timeout,
read/write activity timeout and timout for the entire operation.
Default "entire operation" timeout should be infinity and the
rest just some qualified guesses.
Anyway... I think the current default timeouts are how they
should be.
Regarding the 10 week limit you're mentioning please see the docs
for the Duration type:
http://dlang.org/phobos/core_time.html#Duration
2. PROGRESSBAR
I've added the following to the above example:
client.onProgress = delegate int(size_t dlTotal, size_t
dlNow, size_t ulTotal, size_t ulNow)
{
return 0;
};
When compiled and run we will see the following:
> STOR file.zip
< 150 Opening BINARY mode data connection for file.zip
* We are completely uploaded and fine
* Remembering we are in dir ""
< 226 Transfer complete
* Connection #0 to host 192.168.110.58 left intact
> QUIT
std.net.curl.CurlException@std\net\curl.d(3365): Progress
callback called on cleaned up Curl instance
It looks like onProgress was called when the file was uploaded
and the data connection was closed. What am I doing wrong?
Should I return any other value when ulNow == ulTotal? In code
examples in the documentation onProgress does not return any
value, but it must.
Maybe it would be better to use some simple progress bar by
default when curl.verbose was true? At least something like
this:
static const bStr = replicate("\b", 20);
static float perc;
client.onProgress = delegate int(size_t dlTotal, size_t
dlNow, size_t ulTotal, size_t ulNow)
{
perc = ulTotal ? cast(real)ulNow/cast(real)ulTotal*100
: 0;
writef("%s* %.2f%c uploaded", bStr, perc, '%');
return 0;
};
Thank you in advance for your opinion on this issues.
You should always return 0 if you do not want to abort the job.
What you're describing sounds like a bug and I'll have a look at
it.
I think that including the progress in the verbose mode per
default will generate too much noise.
/Jonas
