I need to review syslogs for over 160 systems monthly, and I am
trying to write a utility to automate bulk downloads from a
custom web service where they are hosted. I need to calculate a
date range for the prior month, add start and end date and a
serial number to the query string for each system, which is easy,
and in a foreach(system; systems) loop in main() I call a
function passing the string url in to download and write a log to
file. For a small number of systems, it works.
My trouble is, using std.net.curl, if I use get(URL) to get the
entire text in a single call and write to file, memory usage
spirals out of control immediately and within 20 or so calls,
gets to about 1.3 GB and the program crashes. If I use
byLineAsync(URL), then foreach(line; range) write the lines to
file one at a time the memory usage never gets above 5MB but it
just hangs always at the 51st call in the loop regardless of what
parameters are in the query string, or how much data I have
downloaded. The program never terminates, even after hours, but I
can't see ANY activity on the process, CPU, mem or network. I can
break my download jobs into <=50 systems (and it seems to work),
but that seems like sweeping something under the rug, probably
leading to future issues.
I'm using the 32bit binary from
libcurl-7.64.0-WinSSL-zlib-x86-x64.zip on the release archive,
and DMD 2.085.0. I've tried curl 7.63 and 7.57 but the behavior
is the same.
Am I doing something wrong or is there some issue with curl or
something else? I'm pretty new to D and I'm not sure if I need to
go right down to raw sockets and re-invent the wheel or if there
is some other library that can help. If I get this working, it
could potentially save myself and many others hours per week.
- Issues with std.net.curl on Win 10 ... cptgrok via Digitalmars-d-learn
-
