On Monday, 25 March 2019 at 16:25:37 UTC, cptgrok wrote:
Am I doing something wrong or is there some issue with curl or
something else? I'm pretty new to D and I'm not sure if I need
to go right down to raw sockets and re-invent the wheel or if
there is some other library that can help. If I get this
working, it could potentially save myself and many others hours
per week.
There is a limit of 50 concurrent messages per thread [1] in
byLineAsync also the transmitBuffers argument takes part in.
So using multiple byLineAsync at same time/thread is going to
block the process, I'm not sure if this is a bug or is by design.
You could use download() in a parallel foreach, something like
this:
import std.stdio;
import std.parallelism;
import std.net.curl;
import std.typecons;
void main()
{
auto connections = 3; // 3 parallel downloads
defaultPoolThreads(connections - 1);
auto retries = 4; // try up to 4 times if it fails
auto logList = [
tuple("dlang.org", "log1.txt"),
tuple("dlang.org", "log2.txt"),
tuple("dlang.org", "log3.txt"),
tuple("dlang.org", "log4.txt"),
tuple("dlang.org", "log5.txt"),
tuple("dlang.org", "log6.txt")];
foreach (log; parallel(logList, 1))
{
HTTP conn = HTTP();
foreach (i; 0 .. retries)
{
try
{
writeln("Downloading ", log[0]);
download(log[0], log[1], conn);
if(conn.statusLine.code == 200)
{
writeln("File ", log[1], " created.");
break;
}
}
catch (CurlException e)
{
writeln("Retrying ", log[0]);
}
}
}
}
[1]
https://github.com/dlang/phobos/blob/master/std/net/curl.d#L1679
