Re: AsyncHTTPClient crashes saying invalid http version
Stop mixing callbacks and async procs. Your `download` async proc finishes before your download operation finishes because you are not awaiting the `download` proc. Just do this: await downloader.downloadFile(node.mediaLink(board, thread), downFileName) Run
Re: AsyncHTTPClient crashes saying invalid http version
> Is this a wrong link? because that doesn't look synchronous.
Ops sorry I had changed the link.
> One important thing to know is that you should never use waitFor in an async
> procedure, use await.
If I use await, the main thread doesn't stop until all the coroutines I create
inside the loop finishes
proc download(node: JsonNode; downPath, board, thread: string):
Future[void] {.async.} =
var
tim = node["tim"].getInt()
ext = node["ext"].getStr()
downloader = newAsyncHttpClient()
downFileName = fmt"{downPath}/{tim}{ext}"
echo("Downloading ", node.mediaLink(board, thread), " as ", downFileName,
"...")
var downloadFuture = downloader.downloadFile(node.mediaLink(board,
thread), downFileName)
downloadFuture.callback=
proc() =
echo("Downloading ", downFileName, " complete")
result = downloadFuture
proc main() {.async.} =
var
boardAndThread = parseBoardAndThread(getRawLink())
downPath = getCurrentDir() & "/" & getRawDownPath()
board = boardAndThread[0]
thread = boardAndThread[1]
if not downPath.existsDir():
echo("Local path not recognized. Creating directory: ", downPath)
createDir downPath
var
webClient = newAsyncHttpClient()
link = getAPILink(board, thread)
j: string = await webClient.getContent(link)
var
jsonArray = parseJson(j)["posts"].getElems()
filteredArray = jsonArray.filter(proc(it: JsonNode): bool = it{"tim"}
!= nil)
downloadSignaledArray = filteredArray.map(proc(it: JsonNode):
Future[void] = it.download(downPath, board, thread))
for it in downloadSignaledArray:
await it
waitFor main() # never gets to finish downloading.
Run
For context, this is how the json looks like:
[https://a.4cdn.org/po/thread/570368.json](https://a.4cdn.org/po/thread/570368.json)
I have to extract `tim` and `ext` from each item in the JArray. Then from the
program argument I have to parse `po` and build the download link like:
`https://i.4cdn.org/po/1546293948883.png`. I need to download from any JsonNode
that has `tim` asynchronously.
Re: AsyncHTTPClient crashes saying invalid http version
> To keep multiple connections open at a time you will need a separate > HttpClient for each. Ah that fixes it. Thanks a bunch.
Re: AsyncHTTPClient crashes saying invalid http version
> Writing a synchronous downloader script was a breeze: > [https://gitlab.com/snippets/1967018](https://gitlab.com/snippets/1967018) Is this a wrong link? because that doesn't look synchronous. You're doing a number of things incorrectly here. One important thing to know is that you should never use waitFor in an async procedure, use await. What I believe is the problem though, is that you are reusing the same AsyncHttpClient instance for multiple concurrent requests. You need to create one AsyncHttpClient instance for each request that you want to be running concurrently. I'm going to create an issue on GitHub to show a better error message for this case. The easiest way to make this work would probably be to split up parseJson(j)["posts"] into x many lists, then run x async procs on each list which create their own AsyncHttpClient and iterate through the list downloading each one-by-one (using await instead of storing each future and awaiting them all).
Re: AsyncHTTPClient crashes saying invalid http version
To keep multiple connections open at a time you will need a separate HttpClient for each. Try moving webClient = newAsyncHttpClient() into the for loop so each call to webClient.downloadFile has it's own client
AsyncHTTPClient crashes saying invalid http version
Hi, I'm still trying to figure out async/await. I am writing a script that
parses a json returned by a forum API and tries to download all the files
described by the json. Writing a synchronous downloader script was a breeze:
[https://play.nim-lang.org/#ix=2gcb](https://play.nim-lang.org/#ix=2gcb)
However I have been struggling with making the downloads asynchronous. First of
all I used the `AsyncHTTPClient` called `webClient`. Then I iterate over the
json nodes to look for parts that I need to construct file download link. Then
I created a `seq[Future[void]]` caled `futures` to store all the futures
returned by `downloadFile()` method. Then I start the loop. Each time I make a
call to `downloadFile()`, I store the future into `futures` seq.
Lastly when the loop finishes I call `waitFor futures.all()` to wait for the
future returned by the `all()` function
([https://nim-lang.org/docs/asyncfutures.html#all%2Cvarargs%5BFuture%5BT%5D%5D](https://nim-lang.org/docs/asyncfutures.html#all%2Cvarargs%5BFuture%5BT%5D%5D))
to yield.
And at the very last I `waitFor` my main function.
import strutils, os, json, strformat, httpclient, asyncdispatch
proc mediaLink(node: JsonNode; board, thread: string): string =
let
fileID = node["tim"].getInt()
ext = node["ext"].getStr()
result = fmt"https://i.4cdn.org/{board}/{fileID}{ext};
proc parseBoardAndThread(uri: string): (string, string) =
var
slashCount = 0
board = newStringOfCap(2)
thread = newStringOfCap(8)
for c in uri:
if c == '/':
slashCount += 1
elif slashCount == 3:
board &= c
elif slashCount == 5:
if c >= '0' and c <= '9':
thread &= c
else:
break
return (board, thread)
proc getAPILink(board, thread: string): string =
fmt"https://a.4cdn.org/{board}/thread/{thread}.json;
proc getRawLink(): string =
if os.paramCount() < 1:
when not defined release: # debugging purposes so I can just call
./threadgrabber without any link and dir
result = "https://boards.4channel.org/wsg/thread/3360760;
else:
quit("Link required. Usage: $threadgrabber ")
else:
result = paramStr(1)
proc getRawDownPath(): string =
if os.paramCount() < 2:
when not defined release:
result = "out"
else:
quit("Download destination required. Usage: $threadgrabber ")
else:
result = paramStr(2)
proc main() {.async.} =
var
boardAndThread = parseBoardAndThread(getRawLink())
downPath = getCurrentDir() & "/" & getRawDownPath()
board = boardAndThread[0]
thread = boardAndThread[1]
if not downPath.existsDir():
echo("Local path not recognized. Creating directory: ", downPath)
createDir downPath
var
webClient = newAsyncHttpClient()
link = getAPILink(board, thread)
j: string = await webClient.getContent(link)
futures: seq[Future[void]]
for node in parseJson(j)["posts"]:
if node{"tim"} != nil:
var
tim = node["tim"].getInt()
ext = node["ext"].getStr()
downFileName = fmt"{downPath}/{tim}{ext}"
echo("Downloading ", node.mediaLink(board, thread), " as ",
downFileName, "...")
var downloadFuture = webClient.downloadFile(node.mediaLink(board,
thread), downFileName)
downloadFuture.callback=
proc() =
echo("Downloading ", tim, ext, " Complete")
futures &= downloadFuture
waitFor futures.all()
waitFor main()
Run
The program crashes with vague error messages, complaining that the http
version is not correct:
[https://pastebin.com/2jXHKbMV](https://pastebin.com/2jXHKbMV)
I'm compiling with `--d:ssl --threads:on` flag. Any pointers?
